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ABSTRACT 


This paper reports the test and evaluation results of the Integrated Damage 
Control Training Technology IDCTT) Trainer. This device — the product of a 
four year advanced development effort — uses interactive courseware which 
incorporates the latest multi-media computer technology to create a realistic damage 
control training environment. The trainer was developed to support a recent 
change in shipboard damage control philosophy called Total Ship Survivability 
(TSS); a concept which emphasizes the simuitaneous repairing of a ship’s combat 
damage while maintaining its ability to fight. 

The new trainer was comprehensively evaluated using performance data and 
survey results collected from students and instructors during a three month test 
period at the Surface Warfare Officer Schooi’s Damage Control School in Newport, 
R.J. Findings from seven different surveys are presented; performance comparisons 
between the conventional trainer and this new trainer are examined; and narrative 
accounts of both students and instructors are reported. The data clearly identify 
and isolate the specific benefits as well as some drawbacks associated with the 
various enabling technologies integrated during the advanced development of the 
prototype. Recommendations about operationally deploying the device are discussed 


and the implications of suggested enhancements are explored. 
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EXECUTIVE SUMMARY 


Recent developments in multimedia technology have made it possible for 
computer generated simulations to create vivid, life-like damage control training 
scenarios. The Navy's damage control community — acquisition agents, trainers, 
and research and development managers — has explored these new technologies to 
produce more realistic training scenarios. One such exploration culminated in 
a multimedia integration of several enabling technologies collectively called 
IDCTT: the Integrated Damage Control Training Technology. 

The Integrated Damage Controi Training Technology (IDCTT) Trainer is an 
interactive video courseware (ICW) medium which incorporates the latest video and 
audio technology to provide realistic damage control training in a simulated 
shipboard environment. IDCTT’s design goal was to provide Damage Control 
Assistants (DCAs) with interactive computer-based damage control scenarios that 
realistically showed the consequences of their damage control decisions in terms 
of their impact on the shipboard environment. This thesis evaluated the extent 
to which this interactive video courseware achieved that design goal. 
Specifically, it validated the effectiveness of the IDCTT Trainer as a training 
tool for Damage Control Assistants (DCAs) and the instructors who train them. 

IDCTT was specifically designed to support DCA training, not in the 
conventional sense, but under relatively recent changes in shipboard damage 
control philosophy. The results of this doctrinal shift has been called the 
Total Ship Survivability (TSS) concept. Total Ship Survivability reasserts the 
World War II concept of simultaneously repairing combat damage while maintaining 
the ship’s ability to fight. TSS is a training concept designed to increase a 
ship’s war fighting capability and the IDCTT Trainer, in turn, ~ is designed to 
deliver that TSS based training concept. 

This paper examined two central aspects of the IDCTT Trainer’s performance: 
its performance in and of itself, and its performance when compared to the 
trainer currently in use, which is called the Damage Control Central Trainer. 


Pinpointing these relatively circumscribed areas enabled the evaluation and the 
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subsequent analyses to produce sufficient data needed to quantify the impact the 
new integrated technologies had upon both student and instructor performance. 

The test and evaluation data collection plan for the device was designed 
to solicit and collect student and instructor inputs, which taken together, 
provided a basis upon which to evaluate how well IDCTT supported DCA training. 
This portion of the study identified system strengths, weaknesses, and design 
features of the prototype which could be improved to enhance its next variant 
and subsequent generations of the product. Also, the measures of student and 
instructor reactions to IDCTT provided a basis from which to gain ins@@meweneo 
the usefulness of the trainer as a fleet training aid. Students and insStpdete: = 
provided this information using short essays and descriptive narrative accounts, 
quantitative subjective rating data, check-off lists of problem features, formal 
surveys, and both structured and unstructured interviews. 

A direct Veempal icon ome llein a4 the DC Central Trainer was done to 
concurrently determine the relative effectiveness of both trainers. The 
evaluation sought to draw comparisons by obtaining student performance scores for 
each trainer type on a variety of comparative dimensions, and on measures of each 
systems’ ability to produce standardized training scenarios. This information 
was obtained during student training sessions conducted in both trainers during 
which performance was carefully graded using a standardized grading protocol. 
Moreover, subjective impressions of user preferences were quantified using 
standardized rating scales designed to further compare and contrast design 
features of both trainers. 

IDCTT’s test and evaluation was conducted at the Damage Control Training 
Department of the Surface Warfare Officer School in Newport, Rhode Island. This 
school provides the only academic training for novice DCAs. Before this study, 
the School’s only source of simulated DCA battle problem training was the 
conventional Damage Control Central Trainer. The new IDCTT Trainer was 
transported to the DCA School and installed in its simulation wing for the 
purpose of the current test and evaluation study. 


Data were collected on three occasions. The first test period, 24 through 


29 March 1994, consisted of preliminary trials designed to validate the planned 
data collection methodology and to evaluate the IDCTT Trainer’s performance 
characteristics. Findings from this pilot test phase were used to modify the 
original test and evaluation data collection plan as well as the actual hardware 
configuration of the IDCTT trainer prototype itself. The validation data were 
collected during the second and third test periods which were conducted 19 
through 21 April 1994, and 20 through 22 June 1994. Thirty-two students and 
seven instructors participated in the evaluation. 

The validation data revealed that the IDCTT Trainer was highly effective 
on the majority of training dimensions specifically evaluated by the test 
methodology. For example, students reported that the IDCTT Trainer was easy to 
operate and extremely useful as a training aid at the Damage Control School. 
They also reported their desire to see the trainer made available for shipboard 
use. When the IDCTT Trainer was directly compared to the DC Central Trainer, 
students reported that the IDCTT Trainer was clearly the preferred training 
method. IDCTT promoted a more rapid acquisition of basic skills and ultimately, 
a higher level of proficiency. Moreover, contrasted to the conventional DC 
Central Trainer, IDCTT induced significantly higher levels of stress and 
motivated students to more actively participate in the damage control scenario. 
Formal statistical tests of the differences between scores from the two different 
trainers revealed that the IDCTT scores were in fact higher than the conventional 
WGm@rainer’s (p < .05). 

Although students and instructors reported that the IDCTT Trainer was an 
effective training medium, they did identify configural and functional 
characteristics of the trainer for potential improvement. For example, some of 
these candidate changes were associated with using the touchscreen monitor to 
input information into the system. Fifty-nine percent of the students reported 
that they experienced some difficulty operating this input device. The screen’s 
primary problem was its low sensitivity to touch and its slow response time. 
Students recommended increasing its sensitivity or simply exploring different 


input methods, such as voice activation for future consideration. The 
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touchscreen design feature was but one of several identified by this evaluation. 
The present report lists other suggested modifications to the system and offers 
substantiation to support the need for reconfiguration in future variants. 

The collective reports from both student officers and instructors who used 
IDCTT’s technologies revealed one broad, simple finding: the trainer was as well 
liked by its users as it was effective, especially when compared to its 
conventional counterpart. As good as IDCTT was however, students and instructors 
freely suggested ways to make it better. The findings and recommendations 
contained in this report, therefore, highlight the indispensable role formal 
test and evaluation methodologies play in the transition of educational products 


from advanced development to service use. 


Saahal 


I. INTRODUCTION 

Throughout American Naval History, damage control has played a key 
role in the success of American warships at sea. The fundamental 
requirement needed to prepare for battle damage is damage control training 
and since it is unreasonable for ships to actually incur self-imposed 
damage, damage control training has come to employ simulations of battle 
damage situations. In the past, these scenarios were relatively crude, 
relying heavily upon human intensive role playing. Recent developments in 
Multimedia capabilities however have made it possible for computer 
generated simulations to create life-like training scenarios. The damage 
control community has adopted these new technologies to produce more 
realistic training scenarios. One such adaptation is a multimedia package 
called the Integrated Damage Control Training Technology (IDCTT). 

The Integrated Damage Control Training Technology (IDCTT) Trainer is 
an interactive video courseware (ICW) medium which incorporates the latest 
video and audio technology to provide damage control training in a 
simulated shipboard environment. This thesis addresses that interactive 
Video courseware. Specifically, it validates the effectiveness of IDCTT 
as a training tool for Damage Control Assistants (DCAs) and its results 
will be used by the Naval Personnel Research and Development Center 
(NPRDC) to justify the use of the IDCTT system as a fleet training aid. 

The focus of the present chapter is threefold. First, background 
information concerning the IDCTT Trainer and the fundamental concepts of 
damage control are discussed. Second, a detailed description of the new 
multimedia components used to create the Interactive Damage Control 
Training Technologies (IDCTT) Trainer 1s provided. A description of the 
user’s interaction in this new medium is included in the discussion. 
Finally, the conventional shore based training technique in use today, the 


Damage Control Central Trainer, is summarized. 


Avs BACKGROUND 

IDCTT was specifically designed to support DCA training, not in the 
conventional sense, but under a relatively recent change in shipboard 
damage control philosophy called the Totai Ship Survivability (TSS) 
concept. Total Ship Survivability employs the World War II concept of 
Simultaneously repairing combat damage while maintaining the ship’s 
ability to fight. After the USS Stark (1987) and USS Samuel BUNRae 
(1988) incidents, it became apparent that the damage control organization 
was able to repair damage necessary to save the ship but in each incident, 
the ship lost the ability to defend itself from further attacks. TSS is 
a training concept designed to increase a ship’s war fighting capability 
and the IDCTT Trainer is designed to deliver that TSS based training 
concept. 

The IDCTT Trainer uses inveessmetc ice from the Battle Damage Estimator 
(BDE) which is a software package that displays the most probable damage 
to a ship after being hit with a particular weapon. Moreover, IDCTT 
interfaces with the Integrated Survivability Management System (ISMS),? 
which provides two dimensional graphic representations of the ship, to 
produce line diagram drawings of damaged areas and systems.* It also uses 
the latest laser disk video, compact disk audio, and computer technology 


to create a realistic training environment. 


i Damage Control Fundamentals 
A basic understanding of what Naval Damage Control is and how it 
1s conducted is needed to understand what IDCTT attempts to accomplish. 
Damage Control is divided into two disciplines; damage prevention, and 


damage containment and repairs. The IDCTT Trainer focuses on damage 


The ISMS system was removed from the IDCTT Trainer after initial test 
results revealed human interface problems which will be discussed later in the 
text. 


The Battle Damage Estimator (BDE) and the Integrated Survivability 
Management System (ISMS) will be described in detail later in the text. 


Nh) 


containment and repairs. This aspect of damage control is defined as 
follows: 
to cut down and localize damage when it occurs by measures such 
as controlling flooding, preserving stability and buoyancy, 


containing and fighting fires, replacing essential structures and 
manning essential equipment (Gritzen, p.372, 1980). 


2: Damage Control Organization 
Since this thesis repeatedly refers to members of the damage 
control organization, this organization will be described in some detail. 
The following provides the reader with a general discussion of the 


operational damage control organization. 


a. Damage Control Assistant 

The Damage Control Assistant (DCA) reports to the Chief 
Engineer who is the Damage Control Officer and responsible for damage 
@oemtrol on the ship. He also reports to the Officer of the Deck’ who is 
the Commanding Officer’s direct representative on all operational matters. 
The DCA uses repair lockers II, III, and V,* which are equipped with the 
appropriate damage control equipment and manned with 20 to 50 personnel, 
to combat shipboard damage. The Combat Systems Maintenance Center (CSMC) 
orovides an interface between the DCA and Combat Systems personnel to 


Seetdinate the alignment’ and repairs of the ship’s weapon systems. 


b. Damage Control Central 
Damage Control Central (DC Central) is a command center from 
which the DCA coordinates shipboard damage control efforts. A support 


organization in DC Central assists the DCA in collecting, processing, 


The Officer of the Deck stands watch on the bridge and is charged with 
coordinating shipboard operations and given the authority to make operational 
decisions in the Commanding Officer’s absence. He is also required to keep the 
Commanding Officer informed of all pertinent shipboard matters. 


*“ Repair II and III are responsible for damage control in the forward and 


after areas of the ship, respectively. Repair V is charged with damage control 
in the engineering propulsion spaces. 


> Due to redundancies built into ships’ weapon systems, there are various 
component configurations that can be used to make a system operational. 
Alignment of weapon systems refers to the selection of available components to 
make a weapon system operational. 


analyzing, and disseminating damage control information. This support 
organization includes Sound Powered Phone Talkers, Plotters and the Damage 


Control Console Operator. These jobs are described below. 


(1) Sound Powered Phone talker. The Sound Powered Phone 
Talker exchanges messages using standard phraseology between the DCA and 
other key stations. These stations include Repair Lockers II, III and V, 


Main Control, GSMC, and the Eradge. 
(2) Damage Control Plotter. The Damage Control Plotter 


plots all emergencies and damage control actions taken using standard 
damage control symbology on the Damage Control Plates.* He also assists 
the DCA by identifying and recommending damage control actions based on an 


analysis of the plot on the Damage Control Plates. 


(3) Damage Control Console Operator. The Damage Control 
Console Operator (DCCO) monitors an alarm display panel which remotely 
indicates damaged systems and compartments. The DCCO also monitors the 
Firemain Alarm Panel and Pump Logic Diagram. This system reports firemain 
pressure, indicates firemain valve positions, and displays which fire 
pumps are operating. The DCCO can open or shut firemain valves and start 


or stop fire pumps from this console. 


Ee The IDCTT System 

The IDCTT Trainer is the result of research conducted to fulfill fleet 
requirements for more realistic training in both the shipboard and 
training command environments. 


The objective of IDCTT is to provide interactive computer-based 
damage control scenarios to Damage Control Officer and enlisted 
students. IDCTT scenarios provide an interactive decision 
environment that show ship damage control decision consequences 
(Ulozas, oe, -lS°Nevemoersisoe 


This technology exploits the ability of computer based training to 


Simulate scenarios that cannot be feasibly replicated in the real world 


° Damage Control Plates are blueprints of the ship that the DCA uses to 
track initial damage and the progression of fires and flooding. Incilu@ed an 
these plates are piping overlays which display the network of firemain, fuel, 


chill water, ventilation, and compressed air pipes and ducts. 


for training purposes. IDCTT attempts to realistically duplicate stressful 

conditions that a DCA would encounter under actual battle conditions. 
Long standing research on decision making under stress suggests that 
those untrained and unprepared to act in these crisis environments 
tend to make poorer decisions than those trained repeatedly in quasi- 
realistic scenarios. At present, such training does not exist in the 
formal school context. (Ulozas, p.1, 15 November 93) 

The IDCTT Trainer manipulates the stress level imposed on the 

student DCA by addressing six different stimulus features listed below. 

e The volume of information conveyed, 


¢ The rapidity with which information is conveyed from various sources, 


- The extent to which stimuli are partially masked by extraneous 
ambient noise, 


- The presence of distractors such as flashing video and loud audio 
alarms, 


*- The onset of unexpected status report inguiries from the ship’s 
Commanding Officer, and 


« Negative feedback from superiors for inappropriate decisions. 

Since the combined effort of the ship’s crew is required to fight any 
damage control problem (damage control problems are generally exercised 
mmeles the ship is at General Quarters (GO)),'’ training time for DCAs is 
limited. IDCTT enables the DCA to realistically exercise decision making 
skills in a worse case scenario, without effecting the ship’s routine. 
Moreover, it provides shore commands with a more realistic training 
environment. 

The IDCTT depends on inputs from autonomous systems to create 
realistic training scenarios and uses the most modern technology to 
display these scenarios to the user. These system inputs, hardware 
requirements, and methods used to implement them in the IDCTT Trainer are 
discussed in the following sections. The first section, "“IDCTT System 
Inputs", describes how the Battle Damage Estimator (BDE) and Integrated 


Survivability Management System (ISMS) interface with the IDCTT Trainer. 


General Quarters is the highest condition of shipboard readiness in which 
the entire crew mans their respective battle stations. 


The second section, "IDCTT Trainer Hardware", describes “cache 


components used by IDCTT to create a realistic training environment. The 


third section, “ZIDCTT Trainer Software", outlines the computer gm... 
useadwto run the LCT iraine:. Finally, the section, “ZOCTT Wee? 
Utilization", describes how the system is manned and operated and a brief 


overview of the battle problem scenario is given. 


his IDCTT SYSTEM INPUTS 

The BDE and ISMS are independent systems which provide inputs to 
the IDCTT. The IDCTT Trainer uses these inputs to provide outputs in the 
form of user training scenarios. The BDE provides the information 
necessary to compose accurate training scenarios that reflect the most 
probable damage for various weapon hits. The ISMS, although an independent 
system, interfaces with the IDCTT hardware to provide various information 
directly to the system. These two subsystems are described moremeuuy 


below. 


a. Battle Damage Estimator (BDE) 
The BDE was developed on a personal computer to display 
probable shipboard battle damage, based on a model called the Ship 
Vulnerability Model (SVM) .* The BDE enables a user to choose a weapon from 


a menu of “eight types, 


and an impact point on the ship. Using these two 
inputs, the BDE then provides damage estimates for the specific weapon 
chosen at the specified hit location. This estimate includes a three 


dimensional image indicating all spaces that would be flooded or on fire. 


Further damage information 1s provided in seven areas: (1) hull, 
mechanical, electrical and combat systems, (2) crew, (3) firemains, 
(4) chill water mains, (5) electrical power panels and cable runs, 


(6) high pressure air mains. and (7) low pressure air mains (Dayvad 


" The Ship Vulnerability Model was developed by the David Taylor Research 


Center with support from Naval Sea Systems Command to provide extremely detailed 
models of the probable damage caused by different conventional weapons on a 
combatant ship. This model is based on actual weapon hit data from the Gulf War 


and shock test data obtained from various types of weapons. 


> Weapon types include Exocet, Harpoon, Stinger, cruise and surface to air 
missiles, contact and influence mines as well as projectiles. 


Taylor Research Center Ship Structures and Protection Department, 
Pez, dume” 19927). 


Information generated by the BDE is used to develop training 
scenarios which are then used by the IDCTT Trainer. Using the BDE allows 
scenarios to be developed from a standardized data base of probable ship 
damage, thus eliminating the developers’ need to approximate what damage 
might occur based on personal opinion. BDE output used to develop training 
scenarios has been successfully demonstrated by the Afloat Training 
Organization (ATO) TSS drills during Refresher Training (REFTRA) since 
1987.' The ATOs use the Total Ship Survivability/Fleet Training Model, 
a model similar to the BDE and also based on the SVM, to provide damage 


estimates for conventional weapon hit scenarios. 
b. Integrated Survivability Management Systems (ISMS) 

ISMS is a NAVSEA project designed to meet the information, 
communication, and command and control needs imposed on DCAs by the added 
complexity of state of the art ships and TSS responsibilities. The goal 
of ISMS is to provide more information to the DCA in a clear compact form. 
Specifically ISMS helps the DCA by: 

- Determining the type and location of weapon effects, 

weecommunicating this information to decision stations, 

¢ Displaying the information, 

+ Integrating the information with the ongoing activities, 

¢« Developing plans of action, 

« Initiating commands, and 

*« Executing the commands by merging traditional, survivability efforts 
with improved communications and computer support (Naval Sea System 
Scand, i>.1, L992). 
The ISMS, which 1s run on a Sun SPARC 10 computer, was integrated 


into the IDCTT system to provide students with all DC plate and piping 


overlay information. Prior to the ISMS system, this information was 


Beenie Atloat Training Organization (ATO) is a shore installation which 


embodies officer and enlisted training experts. The ATO is responsible for 
training ships in various mission areas and quantitatively grading their 
performance during Refresher Training (REFTRA). REFTRA is a periodic 18 month 


training requirement where ships undergo intensive fundamental training on all 
aspects of surface warfare. 


depicted on laminated blueprints to which the DCA would refer for 
information. ISMS now enables students to zoom in on any portion of the 
ship enabling them to obtain exact compartment, access and valve number 
intormaelon: IDCTT also updates the computer generated damage control 
plates with red and blue shadings to clearly indicate fire and flooding, 
respectively. Finally, ISMS uses damage reports from the user’s inputs to 
the IDCTT to update damage control efforts in each affected space using 


Standard damage control symbology. 


25 IDCTT Trainer Hardware 
The IDCTT Trainer integrates the latest multimedia hardware to 
produce damage control scenarios that simulate the shipboard conditions a 
DCA would actually face in the event of shipboard damage. The system 
combines a personal computer, monitors, laser disk, printer, CD rom, 
speakers, and a Sun SPARC 10 workstation to fully emerse the user in a 
realistic simulated shipboard environment. These components and their 


application are described below. 


a. IBM Compatible Personal Computer 
A 486 IBM compatible personal computer is the hardware basis 
of the IDCTT system. This computer provides the scenario event time line 


and updates the training scenario based on student inputs. 


b. Computer Monitors 

The system uses a 21 inch touchscreen monitor and two 15 inch 
monitors. The 21 inch touchscreen monitor is the input device [onmectuq we 
orders pertaining to specific damage control actions. These orders are 
displayed in a menu of damage control options used by the student to 
combat the damage. The monitor’s upper left portion depicts reports from 
various DC Central watchstanders that the DCA receives throughout the 
scenario. 

The two remaining 15 inch monitors are used to provide a 
Damage Control Alarm Panel and a Firemain Alarm Panel and Pump Logic 


Diagram. The Damage Control Alarm Panel alerts the user when space alarms 


have been activated. The Firemain Alarm Panel indicates system pressure, 
firemain value position (opened or closed), and fire pump operations (on 


che ees aan 


c. Laser Disk Player 
The laser disk player is used to provide video images to the 
touchscreen monitor. These images graphically depict shipboard personnel 
providing information to the DCA and subsequently, personnel relaying 


orders from the DCA to the appropriate crew members. 


d. Printer 
A laser printer receives preprogrammed graphic output from 
the personal computer providing the user with Damage” Control Chits." 
These chits are printed out and available for the DCA to review as a 


packup te the voice reports received over the Command and Control Monitor. 


e. CD Rom 
The CD rom provides all scenario audio inputs not included 


in the laser disk output such as alarm sirens, and background noise. 


f. Speakers 
The speakers used in the IDCTT system provide stereo sound 
from two separate input sources. The laser disk player provides scenario 
audio inputs while the CD Rom simultaneously blends background noise to 


create a realistic audio facsimile of a shipboard locale. 


g. Sun SPARC 10 Workstation 
The Sun SPARC 10 Workstation hosts the ISMS system as. This 
system 1S connected to the personal computer through a one way interface 
from the personal computer to the ISMS system. The ISMS system’s graphical 
representations are updated based on the scenario event time line received 


from the personal computer. 


-* Damage Control Chits are hand written notes used in the fleet to 
communicate between the DCA and various personnel. They provide information on 
damage location and efforts to contain and repair the damage. 


3% IDCTT Trainer Software 
The IDCTT scenario is written in the object computer language 


"Ouest". The program uses a next event time advance mechanism called the 
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"event-scheduling approach” .’ This prototype software was developed by 


the Center for Interactive Media (CIM). 


The computer program’s design goal was to develop an automated, test 
question derived model for designing comprehensive, test performance 
driven, remedial instruction which alternately tests and instructs 
until mastery is complete (Surface Warfare Officer School Command 
Report, Problem Description and Needs Justification for Interactive 
Damage Control Training Module, p.3, 4 February 1993). 


4. LDCTT Trainer Urrtazacion 
The IDCTT Trainer was assembled next to the existing DC Central 
Trainer at the Damage Control School. This provided the Damage Control 
School with a computer based training medium unlike any training method 
previously used in the damage control community. To understand how the 
school implemented this system, a familiarization of how the system is 
manned, operated, and what the battle problem scenario entailed is 


desirable. These topics are addressed below. 


a. Manning 
The IDCTT Trainer was designed to be operated by one or two 
users. When operated by a single user, the user assumes the role of a DCA 


who must perform all plotting responsibilities. When two users operate 


i2 The event scheduling approach to simulation modeling is a method where 
future events are explicitly coded into the model and are scheduled to occur in 
the simulated future (Simulation Modeling and Analysis, p.12, 1991). The 
Simulated future is the models best approximation of what would happen in the 
real future, given a specific set of circumstances. For example, if a valve is 
ordered closed, the program acknowledges the order and schedules the valve to be 
closed at clock time plus a previously programmed delay time representing the 
action Of physically closing the valve. 


‘* CIM, an interactive media research facility located in Bethesda, 
Marvland, developed the software for the IDCTT. This software development was in 
direct support of training requirements established by the Naval Personnel 
Research and Development Center and the Naval Sea Systems Command. CIM gained 
expertise in developing interactive software through its ground breaking work on 
the Computer Aided Medical Information System (CAMIS), a program used to train 
medical students in operating room and triage procedures. 


JES 


Ene IDGTT Trainer, One assumes the role of DCA: the other assists as the 


BIiCtErey. 


(i) -One User: When one user mans the system, that person 
monitors ali alarm panels, voice reports, and printed DC Chits. 
Simultaneously, the user must input orders to the Command and Control 
Console and locate damage control information from the ISMS or DC Plates. 
The most time consuming action for the single user is locating valve, 
Ppulknead, fitting, and compartment numbers from the ISMS or DC Plates, 


actions normally done by a Plotter. 


(2) Two Users. With two users assigned, there 1s a 
convenient division of duties: one user takes command as DCA; the other 
assists as a Plotter, providing the appropriate information from the ISMS 
or the DC Plates. For this study, teams of two persons were assigned to 


the trainer. This will be discussed more fully in the methods section. 


b. Operation 

The= program 1S) initaated using A start option on the Command 
Consoie menu screen. The DCA has the option to select essential 
watchstanders and stations from the Command Console menu from which he can 
request information or direct action. Once a watchstander or station is 
selected, a second menu replaces the previous one with a list of options 
that the watchstander or station can perform. This process can be 
repeated until the desired action a student wishes to order is displayed 
on the menu. When an action is chosen, the computer generated audio 
visual representation of the appropriate DC Central watchstander back the 
information the DCA has ordered and relays it to the appropriate station. 

All student inputs to the system (orders) affect the 
Simulation event time line. When an order is given, the required action 
is scheduled into the future. Some events, such as the mine and missile 
hits, are programmed to occur regardless of the user’s actions. Ths 


process provides each student with the same baseline scenario, but 


t- 


Significant variation in the scenario is induced depending on the user’s 


actions. 
es IDET? Seenanio 
The IDCTT scenario starts with the ship going to General 
Quarters. During this evolution, material condition Zebra > aioe 


throughout the ship which the DCA monitors and reports to the Bridge. 
After condition Zebra is set, the ship takes a mine hit aft of the after 


Vertical Launch System™ (VLS memegazime. 


The mine hit produces fires 
adjacent to the aft VLS Magazine, flooding below the waterline, and shock 
damage to various systems. Depending on the actions taken by the DCA, the 
damage may spread, be contained, or repaired. : 

After the DCA has had approximately seven minutes to respond 
to the mine damage, an Exocet missile impacts the ship forward of CIC on 
the 0-1 level.’® The missile inflicts a rupture to the chill watermmeness=m 
that renders the radar system inoperative and causes various fires in the 


vicinity of the blast. Similar to the mine hit, damage may spread, be 


contained, or repaired depending on the DCA’s actions. 


d. Kill Points and Pitfalis 
The IDCTT Trainer scenario uses "kill points" to terminate 
the training evolution when it is determined by the software that the 
Sstudent’s actions would result in a complete loss of the ship. rom 


incorrect actions with iess severe consequences, the program uses 


‘4 Zebra is the code name used to describe the highest material readiness 


condition a ship can maintain. In condition Zebra, all water-tight doors and 
hatches are closed to prevent progressive fire spread and flooding in the event 
of damage. Further, key valves are closed in piping systems, thus breaking them 
into smaller systems, to prevent damage in one area from effecting the entire 
system. 


The VLS Magazine contains the ship’s anti-air and anti-submarine missile 
inventory. There is one VLS Magazine located in the forward and after areas of 
che ship. 

‘ 0-1 level indicates the first deck above the main deck. Continuinmameens 
standard numbering system, the 0-2 level is the second deck above the main deck. 


}— 
tN) 


"pitfalls" to warn the student that an inappropriate action was taken. 


"Kill points" and "pitfalls" are described below. 


(i) 7 shi id Poincs.. The IDCTT Trainer/has five: distinct kill 
points which will terminate the program. Kill points are activated by 
poor or untimely DC efforts or a misunderstanding of the basic DC concepts 
necessary to complete scenario. These five kill points are: 

- The fire consumes a majority of the ship, 

- The Aft VLS Magazine exploded, 

weenil] Water distribution is completely lost, 

* Vital cabling trunk is lost, and 

- The VLS Magazine is flooded without CO’s permission. 

When a program 1S terminated due to kill point 

activation, a message 1s displayed on the Command and Control Console 
specifying which kill point was responsible and the actions taken or 


neglected that caused its activation. 


(27° PPi ties is. The IDCTT Trainer responds to improper 
decisions with negative responses. Four conditions cause a message to 
appear that indicates the DCA committed an error or made a poor decision. 
The tour pitfalls are: 

- The Bridge prompts DCA for a Zebra report, 


- The Chief Engineer informs the DCA that he must obtain permission 
Petror tO Starting a Lire pump, 


- The Commanding Officer orders a status report, and 


- The repair lockers query an incorrect or unreasonable order. 


5. Summary 
The IDCTT Trainer provides students with a fast paced electronic 
training medium that stresses the damage control fundamentals necessary to 
complete a TSS based battle problem. The scenario provides students the 
opportunity to exercise damage control concepts taught in a simulated 
shipboard environment. Stressful conditions that students face while 
performing their job as DCA aboard a ship are realistically replicated. 


In the past, this training was provided by the school’s conventional DC 


Central Trainer. The fundamentals of the present, DC Central Trainer 


method, are outlined below. 


Ge PRESENT DAMAGE CONTROL TRAINER TECHNIQUES 

Team trainers!’ and simulators provide DCAs with experience ommea 
shipboard damage control before they report aboard ship. The Damage 
Control Training Department currently uses trainers and simulators to 
prepare prospective DCAs for scenarios they may encounter in the fleet. 
The trainer used to teach these scenarios is called the DC Central 
Trainer. Seven different lesson topics are taught to each DCA class and 
reinforced through practical problems using the DC Central Trainer. These 
EOpics srnelude- 

«+ Basic DC Central Concepts Simulator, 

«+ Stability DC Central Simulator, 

+ Major Underwater Hull Damage Simulator, 

¢- Main Space Fire Simulator, 

* Chemical and Biological Warfare Defense Simulator, and 
- Radiological Defense Simulator. 

Given that the DC Central Trainer is currently in use by the @seneais 
it provides an established baseline against which to compare the IDCTT 
Trainer's performance. To understand the DC Central Trainer method, a 
description of its objectives and an overview of its implementation at the 


Damage Control School are summarized below. 


i Dc Central Trainer Objective 
For each training scenario, students spend one to two hours in 
the DC Central Trainer combating various subsets of problems. Prior to 
commencing the first simulator training period, a 90 minute classroom 
instruction period familiarizes students with the equipment and concepts 


of simulator team training. 


Team trainers are often used in the Navy to train an individual for a 
specific whatchstation in a group environment that simulates shipboard routine 
and special evolutions. 
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Each scenario has the same enabling and terminal objectives. The 
enabling objective states: 
In the DC Central Trainer, PERFORM the duties of one of the following 
personnel during a major Damage Controi scenario. 
a. DCA 
b. Plotter 
c. Sound Powered Phone Talker (Naval Education and Training 
Command, Surface Warfare Officer Damage Control Assistant (A-4G-0020) 
Beu7se Curriculum Outline, p.4-20, April 1991). 
The terminal objective states: 
In the Damage Control Trainer, DIRECT the DC organization during 
General Quarters and other emergencies by interpreting and solving 
Damage Control Problems. ((Naval Education and Training Command, 
Surface Warfare Officer Damage Control Assistant (A-4G-0020) Course 
CUmrIculum Outline, p.4=-20,;, April 1991). 
The DC Trainer was originally developed as a non-computer based method for 
meeting the enabling and terminal objectives stated in the Course 


Sir ryvculum Out line. 


2. DC Central Trainer Implementation 

Through years of use and modification, the manning and operation 
Semeehne DC Central Trainer has become standardized and only slight 
modifications to these aspects of the trainer were necessary to implement 
@aentTSs based scenario.** Unfortunately, since the DC Central Trainer's 
scenario inventory did not contain a multiple weapon hit TSS based 
scenario, the development of a scenario that could be used for direct 
comparison with the IDCTT Trainer was dictated. A TSS based scenario was 
developed by the author and approved by the Damage Control School for use 
in the curriculum. The scenario provided a direct way to compare the two 
trainers. Table 1 shows the similarities and differences between the 
IDCTT and DC Central Trainers. The key issues of manning, operation, and 


the scenario specifics are described below. 


a. Manning 
The DC Central trainer was designed as a group trainer to 


provide students the opportunity to perform each watchstander’s duties in 


1@ The DCCO position was given added responsibility due to the unique damage 
control organization aboard an DDG-51 Class Destroyer. 


ie 


DC Central. A typical DCA class is divided into groups Oieiive Gen 7 7 
students. Every group performs each of the seven training scenarios once. 
Within the group, members rotate through the different watch stations. The 
DC Central Trainer accommodates a DCA, Plotter, DC Console Operator, and 


two to four Sound Powered Phone Talkers’? depending on the size of the 


group. This arrangement enables each group member to play the role of DCA 


TABLE 1: COMPARISON OF SYSTEM TRAINING FACTORS FOR THE (DCT t= 


CENTRAL TRAINERS 


Pitfalls Same as DC Central Same as IDCTT 


Seenarloe Pisac Lon 


Laster ueroz 
monitoring 


Listeueror —s 
ability to assist 
student 


Student inputs 


Approximately 25 
minutes : 


Directly menirers 
student actions 


Limited to explaining 
what the IDGRY 1s 
expecting 


Drop down menus 
provide students with 


| Option seleceron 





Approximately 90 
minutes 


Physically removed 
from student, monitors 
student verbal inputs 


Able to change 
Scenario to fit 
students needs 


Students can order any 
actions, feasible or 
QKETE 


Scenario complexity Same as DC Central Same as IDCTT 


White noise used to Signifticane amounts None 
distract students 


Speed student must | Fast 


process information 





for at least cone scenaric. 


watchstations during the six week course, 


Although students rotate through the various 


student groups only had one 


exposure to the DC Central Trainer’s TSS scenario. 


Sound Powered Phone Talkers can consolidate or separate phone lines to 


Support two, three or four Phone Talkers. 


16 


b. Operation 

Before starting a DC Central Training session, the instructor 
assigns group members to specific watch stations. The DCA then takes 
charge of his watch team, directs equipment and phone checks, sets up the 
DC Plates and prepares for the scenario to begin. The DCA positions 
himself as if he were in DC Central with easy access to the alarm panels, 
DC Plates, and Phone Talkers. The Damage Control Console Operator (DCCO) 
sits at the Damage Control Console while Phone Talkers man various sound 
powered phone lines. 

The instructor provides all inputs to the watch team using 
Supe ertoted scenario. These inputs are physically displayed on alarm 
panels which indicate fire, flooding, changes in firemain pressure, halon, 
and carbon dioxide release, opening of magazine sprinklers and the onset 
of high temperature alarms. Furthermore, the instructor simultaneously 
plays the role of all watchstations which are based outside of Damage 
Control Central. These roles include the Captain, Bridge, Repairs II, III 
and V, Combat Systems Maintenance Central (CSMC), and any other stations 
necessary to support the scenario. In the present format, instructors 
[teememonitor two or more sound powered phone circuits and the 21 Mc? 
while indicating damage through a remotely operated alarm control panel. 

The DCA receives damage information through the Sound Powered 
Phone Talkers, alarm panels, and the 21MC. As discussed above, all 
information the DCA receives is generated by the instructor. The Plotter 
uses laminated DC Plates to assist the DCA by plotting all the information 
received and actions taken to combat the damage. Unlike the IDCTT 
Trainer, the DC Central Trainer does not have any computer display 


memreors Or DC Chits. The svstem relies on “hard wired" alarm panels and 
\ p 


“° The 21 MC is a hard wired, 10 station, speaker phone system. The net 
includes Combat Information Center, Damage Control Central, Radio Central, the 
Bridge and other key locations throughout the ship. 
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continuous dialogue between the DCA and the instructor to maintain a 


“ontinuous flow Me Intommaticn. 


¢. DC Central Trainer Scenaric 

Although current trainer manning levels and operational 
techniques did not have to be significantly modified for this study, it 
was necessary to develop a new DC Central Trainer scenario. The new DC 
Central Trainer scenario encompassed the same learning objectives, 
management options, kill points, pitfalls, and emphasis on TSS concepts as 
the IDCTT scenario. Since students would be exposed to both methods, 
scenario content had to be similar in difficulty and substance without 
reguiring exactly the same actions needed to successfully complete the 
problem. In other words, if the DC Central and IDCTT Trainer scenarios 
were exactly the same, students would gain insight into the scenario and 
apply to whichever method they were exposed to last. This insight) @enen 
would confound the students’ opinions and performance for the two methods. 

The scenario written for the DC Central Trainer employed the 
same weapon types, one floating mine and one Exocet missile, as the IDCTT 
scenario. As in the IDCTT scenario, the ship goes to GQ and sets material 
condition Zebra. Unlike the IDCTT scenario, however, initial damage is 
caused by an Exocet missile hit forward of the after VLS Magazine. After 
the DCA has been given time to react to the initial damage, further damage 
is inflicted to the ship by a mine explosion on its port sides  =“Bémeene 
convenience of the reader, Table 2 compares the specific damage inflicted 
in the IDCTT and DC Central Trainer scenarios. 

Similar to the IDCTT scenario, damage spreads or is contained 
depending on the actions taken by the DCA. If the DCA reacts poorly to 
the situation, the simulation will end with one of the same kill points 


used in the IDCTT Trainer, at the discretion of Che sinsterucror- 


is 


TABLE 2: COMPARISON OF WHEN MAJOR EVENTS OCCUR IN THE IDCTT AND DC CENTRAL 
MRAINER’S SCENARIOS 


Scenario IDCTT Trainer DC Central Trainer 
Description 


irs t * Fire in aft Generator * Fires forward of the aft 
Weapon Hit Room VLS Magazine 
* Loss of one fire pump | *~ Kiocure GO: porte firemain 


and rupture to stbd loop 
flremain Loop | * Loss of aft radar array 
“WOoss GE are 2adar array * Personnel casualties 
7 “ft looding, art 
* Personnel casualties 


Fires on 0-1 level Fires on second deck 
(above and forward of (below CIC) 
Cie} Flooding in forward 


Chill water rupture compartments 
Personnel casualties * Chill water rupture 
Personnel casualties 





oe Summary 
Performance in the DC Central Trainer provides a standard for 
comparing and determining that of the IDCTT Trainer. Both trainers 
possess the same fundamental goai of training DCAs using simulated 
multiple weapon hit, TSS based training scenarios. The two systems differ 


only in the medium used to conduct this training. 


IIT. METHODOLOGY 


A. OVERVIEW 

This chapter describes the methodology used in the ICW validation. 
The chapter is partitioned into three broad areas. The information 
required to conduct the evaluation is outlined first. Then secondly, the 
method by which data were collected at the test site is given. And third, 


he various statistical techniques used to reduce and analyze the data are 


fel 


O2scusseq. 


B. INFORMATION REQUIREMENTS 
This section discusses the salient information needed to validate the 
effectiveness of the ICW. The validation process itself has been 
described elsewhere, but for the purpose of clarification, the process is 
defined as follows. 
Validating the usefulness of instructional materials involves 
assessing the impacts on the organization in relation to several 
factors. These factors can be classified into several broad 
categories, including: 
* Student achievement; 
- The utility, ease of use, and creativity of the materials themselves; 
- and Integration into the organization’s instructional system (United 
States Naval Health Sciences Education and Training Command, 
Interactive Multimedia Courseware Validation Report, p.2, 3 December 
oe 2 ie 
The present validation effort examined the two central areas reflected in 
the definition cited above. These two areas were: 
¢ The IDCTT Trainer’s performance in and of itself, and 
¢- The IDCTT Trainer compared to the DC Central Trainer. 
Focusing on these areas produced the necessary information to support a 


comprehensive ICW validation. These topics and the information required 


to support their analyses are discussed in the following sections. 


ee IDCTT Trainer Performance Evaluation 
The IDCTT Trainer performance evaluation was designed to solicit 


student and instructor inputs, which taken together, would provide a 
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subjective basis to evaluate the extent to which the IDCTT system 
pported DCA training. The focus of this portion of the study was to 
identify system strengths, weaknesses, and areas which could be improved 
to enhance the system by modifying the prototype. Also, measures of 
student and instructor reactions to this new training medium were taken to 
gain insight into the usefulness of the IDCTT Trainer as a fleet training 
aid. This information was obtained by four relatively straightforward 
data collection techniques. They were: 

« Short essay and narrative descriptions, 

* Subjective rating data, 

¢« Check-off lists of problem features, and 

- Instructor surveys and interviews. 


These methods are each described below. 


a. Short Essay and Narrative Descriptions 
Short essay and narrative descriptions were solicited from 

students after they completed of the IDCTT Trainer training period. 
Respondents evaluated five dimensions of IDCTT Trainer performance in 
essay format. These five dimensions were: 

* Problem features of the IDCTT Trainer,” 

« The performance of the touchscreen monitor, 

- Problems encountered while using the IDCTT Trainer, 

¢« Favorable aspects of the IDCTT Trainer, and 


¢ Aspects of the IDCTT Trainer that were least liked. 
b. Subjective Rating Data 


Subjective rating data were obtained by administering three 
Surveys to students after they completed IDCTT training. Respondents were 
asked to indicate their opinion on various aspects of the trainer’s 
performance by circling a number on an eleven point rating scale. The 


scale was anchored on both ends by appropriate language specific to the 


fe This Wimension provided amplification to the Check=Off 
problem features discussed later in the text. 
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actual question content and the central value indicated a neutral 
response. The three surveys used to evaluate this portion of the 
validation were the: 

« Student IDCTT Survey, 

- User Interface Dimensions Questionnaire, and 

« Source of Workload Evaluation. 

The Student IDCTT Survey was designed to obtain information 
on unique aspects of the IDCTT Trainer. The User Interface Dimensions 
Questionnaire was designed to provide a standard approach in evaluating 
eight key ICW user interface dimensions (Harmon and Reeves, 1993). 
Finally, the Source of Workload Evaluation, a technique developed by NASA, 
was used to assess the relative importance of six factors with regard to 
how much workload students experienced while operating the ICW (Hart, 


1988). The three surveys appear in their entirety in Appendix A. 
c¢. Check-Off Problem Features 


After completing the IDCTT training session, students were 
presented with a list of IDCTT Trainer hardware and software features and 
asked to indicate which feature, if any, caused them difficulty. These 
features were generated from potentiai problem areas identified by the 
system developers at CIM.** The following seven features were evaluated: 

- The operation of the touchscreen monitor, 

+ The clarity of audier -eportc, 

- The ability to find DC Plate information, 

- The acceptability of the speed or volume of information presented, 
- The presentation of the Damage Control Alarm Panel display, 


* The presentation and operation of the Firemain Alarm Panel display, 
and 


* Other features not included in the survey list. 


“- The project development team and the author compiled these topics based 
on their own experience using the IDCTT Trainer. 
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Students were invited to write about any difficulties they experienced on 


these items in space provided on the survey instrument. 


d. Instructor Evaluation 

After completing the training session, each instructor was 
interviewed and asked to complete a survey which evaluated the efficacy of 
the system as a training aid. The interview was open ended, allowing 
instructors to describe their impressions of the system and its usefulness 
as a shore based training aid. The Instructor fOCTT “Survey produced 
ranking data and short narratives. The survey evaluated the instructors’ 
Sia lity to: 

- Use the IDCTT system as a training aid, and 


- To critique students’ performance. 


oR IDCTT Versus DC Central Trainer Performance Comparison 
A direct comparison of the IDCTT Trainer with respect to the DC 
Central Trainer was done to determine the relative effectiveness of both 
trainers. The evaluation sought to draw comparisons by obtaining Selene 
performance scores for each trainer type on a variety of comparative 
dimensions, and on measures of each systems’ ability to produce 
Standardized training scenarios. This information was obtained through: 
« Graded student sessions, ana 
- Subjective rating data. 


These measures are described below. 


a. Graded Student Sessions 

Each method provided the student with the same information 
needed to complete the damage control scenario. Since the scenarios were 
designed to be equally difficult, student performance primarily depended 
on the medium in which they operated. A standardized scoring system was 
developed for assigning grades to student performance in each trainer. 
When compared, these scores provided a baseline measure of the IDCTT’s 
effectiveness against the present standard; that is, performance in the DC 


Se weral Trainer. 
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Prior to the present study, DC Central Trainer sessions were 
ungraded. The Damage Control School's policy was to simply expose DCAs to 
damage control situations and not formally grade them. To facrie ie 
gJuantitative basis on which to compare students’ performance between the 
two methods, specified grading criteria were developed with assistance 
from instructor experts at the Damage Control School. The criteria were 
modeled after Fleet Exercise Publication Number 4 (FXP-4)* grading 
standards which are used to grade damage control organizations in the 
fleet. The grading criteria developed for the present evaluation 
emphasized basic damage control principles, TSS concepts, asset 
management, and common sense. The same grading criteria, a complete copy 
of which 1s provided in Appendix A, were used for both the IDCTT and DC 


Central Trainers. 


b. Subjective Rating Data 
Subjective rating data that compared aspects of the two 
systems were obtained using two separate surveys. The two surveys were 
the: 
« IDCTT Versus DC Central Trainer Comparison Survey, and the 
- Scenario Topics Ranking Survey. 
The Trainer Comparison Survey requested students compare the two systems 
across eleven different dimensions. The Scenario Topics Ranking Survey 
measured students’ perceptions regarding the consistency with which each 
trainer met the specified damage control learning objectives. These two 
surveys are described below and are included in Appendix A for the 


iWeeres tea meagdame 


(1) IDCTT Versus DC Central Trainer Comparison (597-72 
After students were exposed to both training environments, they completed 
a survey which was designed to evaluate the IDCTT Trainer with respect to 


the DC Central Trainer. Eleven system characteristics were presented on 


°° FXP-4 provides Afloat Training Organizations with a grading criteria for 
various damage control problems while providing ships with a training guideline. 
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a bipolar preference scale anchored on one end by the IDCTT and on the 
other by the DC Central Trainer. A "six" on the scale indicated "no 
preference" between the two methods. Students circled the number which 
corresponded to the degree they felt one method outperformed the other. 
This item series attempted to determine which of the two systems better 
met the following eleven criteria: 

- Simulated the shipboard environment more realistically, 

- Enabled instructors to provide complete post scenario debriefs, 

¢ Produced the greatest level of stress, 

° Allowed instructors to monitor student’s performance, 

* Prepared the student for actual shipboard emergencies, 

- Updated student inputs more easily, 

¢ Provided scenario information closely resembling shipboard methods, 


+ Provided more effective teaching environment to exercise damage 
Gemcro! skills, 


- Promoted greater learning in the time allotted, 
¢ Preferred training method, and 
- Stimulated the student to perform. 
Responses from this bipolar ranking series was used to determine how the 


two methods compared across the range of capabilities listed above. 


(2) Scenario Topics Ranking Survey. The Scenario Topics 

Ranking Survey was administered after students completed each training 

method tc measure the extent to which each trainer delivered a 

standardized scenario. The survey listed a series of fundamental damage 

control actions needed to successfully complete a damage control problem. 

Students rated the extent to which each fundamental topic played a role in 
the battle problem delivered by the IDCTT and DC Central Trainers. 

The Scenario Topics Ranking Survey yielded two measures. 

First, an ordinal ranking of the 13 damage control fundamentals each 

scenario emphasized was made based on the median scale value from the 


subjective rating responses. This ranking was used to determine if the 
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two trainers emphasized the same damage control fundamentals, and using 
the median responses, to what degree. Second, the interquartile range 
(IOR) from each trainer’s rating data was compared to determine how much 
the student ratings varied across the two trainers. These data 
highlighted the extent to which each system consistently emphasized the 


same learning objectives from the perspective of the student. 


Ge DATA COLLECTION METHODS 

The IDCTT Trainer validation study was conducted at the Damage Control 
Training Department of the Surface Warfare Officer School in Newport, 
Rhode Island. This school is the sole source of academic training for 
novice DCAs. Before this study, the school’s only source of simulated DCA 
battle problem training was the DC Central Trainer. The new IDCTT Trainer 
was transported to the DCA School and installed in it’s simulation wing 
tor the purpose of Enis Validation sendy. 

Data were collected on three occasions. The first test period, 24 
through 29 March 1994, consisted of trials used to test thom 
collection methodology and the IDCTT Trainer’s performance characteristics 
in a field environment. The second and third test periods, conducted 19 
through 21 April 1994, and 20 through 22 June 1994, were used Gomes 
the actual validation data. The three test dates were scheduled during 
the last week of each DCA class to ensure students had the knowledge 
necessary to complete complex damage control scenarios, such as those 
which would be presented in the IDCTT. 

As discussed in the previous section, the data used in this validation 
was collected by a series of surveys, interviews, and graded performance 
evaluations. Data were collected after students completed training in 
each type device. Comparison data; that is, data that were wWil@ameeme 
used to compare and contrast the technical features of both training 
devices, were collected upon completion of the entire training evolution. 


Instructor data were collected after the student training sessions were 
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completed. Table 3 shows the type of data solicited at various stages of 
the data collection period. The following sections describe student 
assignments for training, survey administration, student scoring criteria, 


and the pilot test period used to finalize the validation methodology. 


TABLE 3: DATA COMPLETION POINTS DURING THE EVALUATION STUDY PERIOD 


a —_ ee ee — 





Sources of Data Data Collected 


BDGrT trainer ke Studene ©DCim Trainer Survey 
Scenario Topics Survey (for IDCTT) 
Source-of-Workload Evaluation 
User Interface Dimensions Questionnaire 
Student Grade Sheet 


DC Central Trainer . Scenario Topics Survey (for DC Central 


Trainer) 
Student Grade Sheet 


Both Trainers . Student IDCTT vs DC Central Trainer 
Comparison Survey 
i= 7 = = ee | 
Student foie hyvsteucrersIPeT P Ssunvey 
Participation 2. Instructor Interviews 


aes Student Assignments 

Students were in thelr final week of damage control school when 
they participated in the validation study. Three class days were 
scheduled for each of the data collection periods. During each of these 
periods, students were randomly assigned in groups of two for the IDCTT 
Trainer, and groups of five to seven for the DC Central Trainer. For each 
of the test periods, a class of 30 students were scheduled to participate 
in the study. However, due to scheduling conflicts and time limitations, 
only a portion of each group completed the entire evolution. The 
following guidelines were used to schedule the students through the two 
training methods: 


- Half of the students were assigned to the IDCTT Trainer first while 


the other half used the DC Central Trainer. This procedure ensured 
each trainer had the same number of students exposed to its style 
first. 


+ Each of the two students assigned to the IDCTT Trainer was given one 
practice session and one qraded session. Furthermore, The two 
students in the group exchanged duties as DCA and Plotter for a total 
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of two practice sessions followed by two graded sessions in one 
EvaIningsperiled- 
* The following four position sequences were randomly assigned to 

students pemfesmingw@in the (DCRieTrainer: 

1. DCA-Plotter-DCA-Plotter, 

2. Plotter-DCA-Plottem=DCA; 

3. Plotter-DCA-DCA-Plotter, and 

4. DCA-Plotter-Plotter-DCa. 


*- The student DCA for the DC Central Trainer was randomly selected from 
the group of five to seven students for each training group. 


¢ Student groups were scheduled for 90 minute training periods for each 
training method. 


Qe Survey Administration 
Before a survey was distributed, its instructions were read 
aloud, and students were encouraged to: 


« Seek clarification on parts of the survey they did not understand, 
and 


+ Comment on items they felt were important, but were not inc Tie ae 
the survey. 


The order in which students received the six different surveys depended on 
which training method they were exposed to first. Surveys were 
distributed according to the guidelines shown in Table 3. Each survey was 
given to the students immediately after a training session (IDCTT Trainer, 
DC Central Trainer, or Both). Both the survey coordinator and instructors 
were avallable to clarify any misunderstandings the students had on the 
surveys. Instructor surveys were distributed after students completed 


their training sessions. 


z\ Student Scoring Criteria 
Student performance was monitored and graded in both the IDCTT 
and DC Central Trainers. Grades were assigned based on students’ 
demonstrated level of proficiency during the 90 minute training period. 
Grade assignments were standardized using the Student Grade Sheet criteria 


described earlier. This grading procedure provided student data for one 
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graded IDCTT Trainer session and one graded DC Central Trainer session.*4 
It should be noted, however, that although the same grading criteria were 
used for both trainers, the ability of the instructor to monitor the 
Students‘ performance varied between trainers. The following sections 


describe how instructors graded the students for each trainer type. 


a. IDCTT Trainer Grading Protocol 

Each of the two students assigned to the IDCTT Trainer 
completed one practice session as the DCA and one as the Plotter. 
Students were encouraged to ask questions about any aspect of the trainer 
they did not fully understand. After their questions were answered, and 
they reported feeling comfortable using the new trainer, each student 
completed one IDCTT Trainer scenario as the DCA and was assigned a 
quantitative grade that reflected their level of proficiency. Instructors 
observed the students’ progression through the scenario from inside the 
IDCTT Trainer module room. This enabled instructors to visually monitor 
Student input into the IDCTT’s Command Console, and to observe the 
Students’ reactions to the various stressful situations. From this 
vantage point, instructors assigned grades on a scale from one to 100, 
based on the criteria specified on the Student Grade Sheet. Points were 
deducted depending on the percentage of each grading topic not 


completed.?° 


b. DC Central Trainer Grading Protocol 
All students who participated in the evaluation received 
Beaming throughout the DCA School’s curriculum in DC Central Trainer 
operations. Thus, Students were assumed to be proficient in this 


particular trainer before the day of the actual test session. Student 
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School. Thus, students were already skilled in the use of this method. 


for post scenario debriefs, 


Before the validation test date, students had completed seven different 
DC Central Trainer scenarios which were required for graduation from the DCA 


2° It should be noted that although the IDCTT Trainer has a playback feature 


Seudent scheduling - 
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tHis feature was not wsed due to time constraints in 


groups were scheduled for 90 minutes of DC Central Trainer simulation 
training using the sfTotal Ship Survivability based battle (an25i-7. 
Students in this group were randomly assigned to the DCA and other 
positions by the instructor. The simulation was run once, and took the 
entire 90 minute training period to complete. As with the IDCTT 3227 
instructors graded student’s performance based on the standardized grading 
Craeeria. 

One aspect of the DC Central Trainer that deserves comment; 
that is, in this trainer, the instructor is physically isolated from tne 
students during the exercise. Since only one instructor was assigned to 
each group, there was no direct means to evaluate the students’ 
performance except from the inputs the instructor received from the other 
side of the wall. Unlike the IDCTT Trainer which recorded the entire 
training session and was designed to accommodate an i1nStruclOn mame] 
room, an error in the DC Central Trainer was lost if it was undetected in 
the voice communications. Generally, this arrangement would bias test 
results toward better performance in the DC Central Trainer because some 


mistakes would go unobserved causing higher performance scores. 


4. Data Collection Pilot Test Period 
Initially, the data were scheduled to be collected during 24 
through 29 March 1994. This test period was the first time IDCTT was used 
outside the development facility. This pilot test period was used to test 
the system hardware and software performance in its intended operating 
environment, and to verify and standardize the data collection 


methodology. 
a. Schedule 


As discussed above, the validation series lasted three days, 
during which 30 students were scheduled to participate in the study. All 


30 students completed the DC Central Trainer portion of the test sequence, 
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Eeieeecue CO Trequent ~“crashes" of the IDCTT Trainer, only l2 students 
completed the IDCTT evaluation phase.*® 

The original test plan required each student to complete two 
practice sessions in the IDCTT Trainer followed by a third session which 
was graded. This procedure took about 90 minutes of single user IDCTT 
Trainer time per student; approximately the same amount of time as one DC 
Central Trainer scenario. Due to the limited time students were available 
for testing, this requirement was revised to one practice session followed 
by a graded session. Further, original test design required the student 
to operate the IDCTT Trainer in the single user mode, acting as their own 
plotter by manipulating the ISMS. This procedure was changed because 
students spent excessive time trying to find information using the ISMS, 
which distracted their attention from the main battle problem. To offset 
this attentional diversion, a second student was added to operate the ISMS 
and perform plotting duties. 

The 30 student participants completed their scheduled IDCTT 
practice runs, but unexpectedly, this took the entire allocated testing 
period. The school was unable to accommodate the allocation of further 
testing time for the purpose of this pilot study due to its existing class 
schedules. The school permitted continued testing for the graded 
sessions, after hours, on a voluntary basis. Twelve of the 30 students 


volunteered to participate in this follow-on testing. 


b. ISMS Removed 
For the graded sessions, ISMS was dropped from the IDCTT 
Trainer system and replaced with DC Plates. ISMS was deleted for three 
reasons: 


¢ Students had not been previously exposed to the ISMS system, thus 
requiring them to learn the two systems simultaneously. 


-° During the pilot test period, a system "crash" occurred approximately one 
time in every six tralning sessions. Once a "crash" occurred, the system had to 
be soft booted to reinitialize the training program. This procedure lost the 
previous scenario, thus requiring the student to start from the beginning. 
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- Students could not locate specific valve numbers from the piping 
diagrams due to the small scale representation. 


- Software interface problems between the 486 computer and the ISMS 
system frequently caused the system to lock-up. 


The ISMS system therefore was deleted from the IDCTT Trainer throughout 


the remainder of the validation study. 


c. IDCTT Trainer System Improvements 
Upon completing the pilot test period, the IDCTT Trainer was 
shipped to CIM for software updates based on the information obtained 
during the jpmileteerestic. These updates will be discussed later, but 
suffice it to say here that the revised version eliminated system crashes 
and improved the user interface and increased the likelihood that students 


would complete the scenario. 


qd. Summary 
From the information obtained during the pilot testieemmedy 
the data collection methodology actually used was only slightly medma ya 
Further, the prototype IDCTT Trainer was also modified based on 
information collected during the initial tests. After these changes were 
implemented, the IDCTT Trainer version was not modified during the 


remainder of the study. 


De DATA ANALYSIS METHODS 
Upon completion of each test trial, all data were sorted and placed 
into individual student data packages. A student file contained the four 
data records: 
- Short essay and narrative descriptions, 
* Subjective Rating Data, 
- Frequency data,’’ and 


- Performance grades 


“’ Prequency data were collected on the percentage of students who felt 
various features of the IDCTT Trainer required improvements. 
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These data were then compiled to accommodate the statistical analyses 
employed by this study. These statistical methods are described in the 


following sections. 


1 Short Essay and Narrative Descriptions 
Short essay and narrative descriptions were included in the 
validation study to augment the rating data and frequency responses. 
Because of the qualitative nature of the narratives, summary statistics 
could not be computed, but similar student responses were categorized, 
then summarized and listed for each subject. These listings provided a 
means, although subjective, to isolate patterns in student responses that 


further explicated their quantitative responses. 


ah. Rating Data 
The rating data were analyzed using the Method of Equal-Appearing 
Intervals (MEAIS) (Thurstone and Chave, 1929). Using this method, the 
scale data obtained from tne student questionnaires were compiled and 


placed in summary tables similar to that shown in Table 4. 


TABLE 4: EXAMPLE MEAIS DATA MATRIX 


Sorting Categories 
5 6 7 8 9 10 rl 
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For each statement, three rows were used to display summary statistics 
from the student data. The first row (f), indicated the total frequency 
that students circled each number. The second row (p), displayed the 
frequency as a proportion to the total number of students. The final row 
(cp), cumulatively summed the proportion of frequency that had occurred 
through the numeric values. 

If the median of the distribution of judgments for each statement is 

taken as the scale value of the statement, then the scale values can 


be found from the data arranged in the manner of Table 4 by means of 
the following formula: 


S=L+(.50>) Paeay a 


where S = the median or scale value of the statement 
L = the lower limit of the imeerval ines ene ere 


median falls 
2p, = the sum of the proportions below the 
interval in which the median falls 
Dp, = the proportion Within the Interval in whieh 


the median falls 
i = the width of the interval and is assumed tc oA 


equal to 1.0 (Bdwards7.pa/ 2 1957). 

The interquartile range provided a measure of the variability in 
the distribution of student responses. The interquartile range (Q or IQR) 
provided a numeric value representing the range of numbers in which the 
middle 50 percent of the scale judgments fell. To find Q, the 25th and 


75th percentiles were calculated and subtracted using the following 


formulas: 
@--Lt (25) Oy Dae 


where C,, = the 25th percentile of the statement 

L = the lower limit of the interval in which the 25th pereeneate 
falls 

2p, = the sum of the proportions below the interval in which 

the 25th percentile falls 

p, = the proportion within the interval in which the 25—en 
percentile falls 

1 = the width of the interval and is assumed to be equal to 1.0 
(Edwards, p89, 1957); 


Ce=l@75— epee 


where C;, = the 75th percentile of the statement 
L = the lower limit of the interval in which the 75th percentile 


falls 
>p, = the sum of the proportions below the interval in which the 


75th percentile falls 
p, = the proportion within the interval in which the 75th 


percentile fails 
1 = the width of the interval and 1s assumed to be equal to 1.0 


(Edwards) oo mela sie 


QO=C)5-C35- 
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Once the scale value (S) and the interquartile range (Q) were calculated 
for each statement, summary statistics indicating students reaction to the 


various items were compiled for further analysis. 


Ba Frequency Data 
Frequency data were analyzed using the proportion of students who 
responded to each check-iist item. This summary statistic was obtained by 
summing the number of students who checked each item and dividing by the 
total number of students in the study. This information provided a 


percentage of students who agreed on each checklist item. 


4. Performance Grades 
The significance of the differences between paired performance 
grades for the IDCTT and DC Central Trainer was evaluated using the Large- 
Sample Wilcoxon Signed-Rank Test for a Paired Experiment (Mendenhall, 
1990). This test determines if the relative frequency distribution for 
the two sets of grades are identical or different. 
The following summarizes the testing procedure. 


To carry out the Wilcoxon test, the Differences for each of the 
Paired scores is calculated. Differences equal to zero are 
eliminated. The rank of the absolute values for each of the 
numbers 1s determined, assigning a 1 to the smallest, 2 to the 
second smallest and so on. The rank sum is calculated for each 
the positive and negative differences. The positive value of 
these two calculations is used to calculate the z value from the 
normal curve and is used as the test statistic the test 
Peaeiceic.- Thus test statistic 16 {2hen compared aqainst ehe 
zZ value for the appropriate significance level desired. This 
comparison is used to determine if the null hypothesis, that the 
two frequency distributions are the same, should be accepted 
(Mendenhall, p.680, 1990). 


The results of this test provided a means for evaluating whether the 


trainers were equally effective as a student training aid. 


BE. SUMMARY 

The pilot test period indicated that the validation test plan was 
mPeeactacable, with only slight modifications. As a result of the pilot 
test, the prototype trainer underwent only minor software updating, and 
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See Appendix C for the z value equation. 
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the ISMS system was dropped from the IDCTT Trainer. Accordingly a2 
summary of the validation methodology is displayed in Figure 1 for the 


convenience of the reader. 


Data Processing Plan 
Topic Instrument Reduction Measure Analysis 


Com > ari Ut) 
Performance 
Scenario Topics S and Q 
Rankings Values. 













Figure 1: Data Processing Plan 


B16 


Liat. ReEsubts 

This chapter presents the results obtained from the validation 
effort’s experimental group. The validation experimental group consisted 
of 32 students from two separate classes. As previously stated, data were 
collected on a total of 32 students; 22 students who were tested between 
19 and 21 April 1994, and 10 students who were tested between 20 and 22 
June 1994. Students in this test group represented a cross section of the 
average classes‘ fleet experience level, and damage control aptitude 
fgullian, 1994) . 

The data are presented in two sections. The first section presents 
the results of the IDCTT Trainer performance evaluation; that is, the 
evaluation of the IDCTT Trainer alone. The second section provides the 
results of the IDCTT versus DC Central Trainer performance comparison. No 
attempt is made to provide a substantial discussion of the results in this 


chapter. That will take place in the next chapter. 


A. IDCTT TRAINER PERFORMANCE EVALUATION 

Again, data were collected from 32 students undergoing instruction and 
the eight Damage Control School instructors to evaluate the IDCTT Trainer. 
These data were comprised of short essay and narrative descriptions, 
rating data, check-off problem features, and instructor evaluations. 


These results are presented in the following four sections. 


Le Short Essay and Narrative Descriptions 
Short essay and narrative descriptions were solicited from 
students using the Student IDCTT Trainer Survey. Each student responded 
to five essay questions. The first question which asked students to 
identify and explain any IDCTT Trainer feature which caused them 
difficulties while interacting in the IDCTT training environment, are 


summarized in Table 5. The final four questions, which also dealt with 


cy; 


IDCTT features, are similarly summarized in Table 6.°° A compiUajeneqmos 
all the students’ responses is included in Appendix B. 


TABLE 5: STUDENT RESPONSES TO A LIST OF IDCTT TRAINER FEATURES 2 SS 
THEY WERE ASKED TO EXPLAIN WHICH FEATURES CAUSED THEM DIFFICUET tee 
OPERATING THE SYSTEM 


a 





Feature Comments 


Touchscreen . Touchscreen is not sensitive to the touch and 
Monaiiee responds slowly to student inputs. 
The screen displayed options which can not be 
activated or serve no purpose. 
Option selections buttons are too close together 
causing accidental inputs imo Che MDCT T 
Trainer. 
Students were unfamiliar with which option 
selection was necessary to initiate specific 
aCe LONs: 


Understanding . Background audio track contained clearly 

Audio Reports erroneous information. 
Unable to ask computer simulated watchstanders 
to repeat previously given information. 


Finding DC . Student unfamiliarity with thé design Gf Gane 
Plate DDG-51 Class Destroyer made locating information 
Information from it’s DGmpileates diatieulc- 
Lack of practice in using DC Plates made it 
difficult to find, information irom, #hesDCeRioee— 
within the limited time allotted in the quick 
pace of the IDCTT scenario. 


Speed or volume . The scenario was too fast for beginners, the 
of information scenario pace should be adjustable to meet the 
presented student’s level of expertise. 


Firemain Panel . The firemain Alarm Panel Screen was too small 
and firemain making it difficult to read valve numbers and 
valve and pump labels. 

operations 


Other features . Unfamiliarity with the system made it initially 
net lasted | difficult totoperaincs 
System 1s unresponsive to student inputs to 
activate all relevant valves or investigate and 
combat damage in some of the damaged 
compartments. 
Unable to obtain specific information on 
integrity of the chill water system. 


°° Summary statements are based on a compilation of written and oral 
responses selected as a representation of the students’ responses. A complete 
listing of all student responses is included in Appendix B. 
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TABLE 6: 





Question 


How can the 
touchscreen 
ytontrol panel 
| be improved? 


What problems 
did you 
encounter while 
Hesinee the IDCTT 
Trainer? 


' What aspects of 


Efe LDCTT 
Trainer did you 
like the most? 


i} 
What aspects of 
tie LDCTT 
fimainer did you 
like the least 


SUMMARY OF STUDENTS’ 
SGHUDENT IDCTT TRAINER SURVEYNARRATIVE*QUESTIONS 





SHORT ESSAY AND NARRATIVE RESPONSES TO 


ee a a —_= 





Samed fl 


Comments 


Increase the size of the number pad on the 
touchscreen or only use the computer keyboard. 
2. Increase the touchscreen sensitivity. 


| 3. Use a mouse to select options or switch to voice 


activation technology. 
4. Eliminate the touchscreen options which are not 
activated. 


1. Inputing information in the Command Console 
touchscreen monitor. 

2. Unfamiliarity with the DDG-51 Class Destroyer 
made it difficult to visualize what the 
information received meant. 

3. Not fully understanding what each button option 
would do stemming from inexperience in using the 
system. 

4. Inexperience in making reports to the Commanding 
Officer, while the system provided no clear 
means to report to him. 

5. Little or no knowledge of the chill water 
system. 


6. System occasionally lTocked-up for no apparent 
__ reason. 


1. The trainer was extremely realistic. 
2. The amount of stress induced by the system. 


| 3. Ability for one person to train without the need 


for an entire watch team. 


\ ite Charitysor inlormmacion the 2DeTr Trainer 


presented. 


| ©. Fast pace and audio-visual presentation of the 


Lil Ome: Lon. 

6. Printed damage control chits. 

7. Enables the user to observe the consequences of 
their actions while providing the option to 
repeat the scenario until it is done correctly. 


Only one scenario for only one ship class. 

2. Difficulties experienced while inputting 
information into the touchscreen monitor. 

3. The emphasis on the chill water system when the 
student users were not trained in this area. 

4. Reasons for “kill point" activation were too 
vague. 


Bis Subjective Rating Data 


Subjective rating data were collected using the Student IDCTT 


Trainer Survey, 


Source-of-Workload Evaluation, 


and User Interface 


Dimensions Questionnaire. Each survey measure solicited different aspects 


of the trainer by responses on a scale from 1 to 11. The scale values and 


Sy, 


interquartile ranges (IQR) for these surveys are presented in Tables 7, 8, 


and 9. 


TABLE 7: SCALE VALUES AND INTERQUARTILE RANGES OF ITEMS EVALUATING STUDENT 
IMPRESS IONS OF NINE .PESIGh ASEECTS2er inky tear tar Rare 


| Question Scale Value | TGR 
How easy was the system to operate? Sa 2 1.37 
(Ditfacuitwekasy) 
Relevant information options on VANE, 3 00 
touchscreen monitor? (None..All) | 
How easily touchscreen allowed pro) |: 4255 
information Input? “Pw rercute nas.) 
Was the scenario too difficult? 
(DItTfi Teulreesasy) 


Was the scenario easy to understand? 

(Confusing. .Clear) 

Was the scenario’s pace too slow? bec i ie 
(Fast. .Slow) 


Was the scenario realistic? 
(Unrealistic..Realistic) 


Usefulness as a DC School Training aid? 
(Worthless. .Useful) 


Usefulness as a shipboard training aid? 
(Worthless. .Useful) 





TABLE 8: SCALE VALUES AND INTERQUARTILE RANGES OF ITEMS EVALUATING 
STUDENTS IMPRESSION OF EIGHT INTERACTIVE COURSEWARE INTERFACE DIMENSIONS 










Statement Scale Value IQR 


Ease of use (Difficult. .Easy) 
Navigation through ICW (Difficult. .Easy) 


Cognitive load (Unmanageable..Manageable) eel 


Mapping (None. . Powerful) 
aos of space compatibility 8.93 1.94 
(Incompatible. .Compatible) 

Information presentation (Obtuse..Clear) 


Media Integration 9.77 1.94 
(Uncoordinated. .Coordinated) 








Overall Functionality Dinbs 1 soe 
(Dysfunctional. ._Highige® Functional) 


40 


TABLE 9: SCALE VALUES AND INTERQUARTILE RANGES OF ITEMS EVALUATING 
STUDENT IMPRESSIONS OF SIX HUMAN WORKLOAD DEMAND DIMENSIONS 


— ~ ~ 
Dimension TOR 
Mental Demand (Low. .High) 







: 1.84 

| Physical Demand (Low. .High) ay i 
| Temporal Demand (Low. .High) i 2G 
Performance Demand (Low. .High) 25 6 

Effort Demand (Low. .High) | | o'350 1 79 





Pruet ratwonedow . aHaghy) 6. 50 





34 Check-Off Problem Features 
Students were presented with seven IDCTT Trainer features and asked to 
indicate which feature, if any, caused them problems. The number of 
student responses and the percentage of students who identified each topic 


as an area where improvements should be made are provided in Table 10. 


TABLE 10: NUMBER AND PERCENTAGE OF STUDENTS WHO IDENTIFIED CHECK-OFF SHEET 
ITEMS AS FEATURES OF THE IDCTT TRAINER WHICH COULD BE IMPROVED 
IDCTT Trainer Feature Number of Percentage of 
Students Who the 32 
Checked Each Students Who 
Feature out of Checked Each 
32 Students Feature 


Operating touchscreen monitor 


Understanding audio reports 


Locating DC Plate information 


Speed/Volume of information 
DC alarm panel display 
Firemain panel and pump operations 


Otmer features not listed 





a. Instructor Evaluations 
Short essay and narrative descriptions, as well as rating data 


were collected from the Damage Control School instructors using the IDCTT 
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InstruGceor Suzver A summary of the instructor comments answers is 


provided in Table 11. Table 12 provides the results from the instructor 


rating responses and interquartile ranges from the five rating categories 
solicited. Appendix B contains a complete listing of all the instructors’ 


short essay and narrative descriptions. 


TABLE 11: SUMMARY OF INSTRUCTOR SHORT ESSAY AND NARRATIVE RESPONSE 
QUESTIONS FROM THE INSTRUCTOR IDCTT TRAINER SURVEY ON THE OPERATICNSGr aa 
A Bd Dl oN le 


Question Comments 


Abilwey of “these Ttratticr temprecens 
identical scenarios toemeachwDGA senaence 

The level of realism that the IDCTT Trainer 
induces. 

Ability tePobjectively critYoue the Beas 
actions. 

Reinforces basic and advanced damage control 
‘concepts. 


What aspects 
gid vouslake 
about the, [DG 
Mreineraror 
teaching damage 
SOniae. | 
problems? 


The program did not run real time, rather it 
moved more like a video game. 

Touchscreen monitor slowed student responses. 
Student unfamiliarity with the system. 
Extensive pre-brief in classroom required to 
“fully prepare students for IDCTT. 


What problems 
did you 
encounter while 
using the IDCTT 
Trainer as an 
l@striuct tonal 
aid? 


= i — a so | a —#=$ — ee = - —— 





Provide more detailed feedback from the repair 
lockers in response to incorrect or improper 


What aspects of 
the I ber. 


Trainer would 
you like 
changed? 


orders. 
Change the method of inputing orders from the 
touchscreen to voice recognition. 


CSMC should have its own repair team to isolate 
and correct chilli water problens. 
Increase the number of scenarios. 











What benefits 
do you envision 
from the use of 
the IDCTT 
Trainer at the 
Damage Control 
School? 


May be used as a final simulation to objectively 

determine a students’ ability to operate under 

stressful conditions. 

Better preparing DCA students for the pressures 
aa problems associated with actual damage 
control scenarios in the fleet. 
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TABLE 12: SCALE VALUES AND INTERQUARTILE RANGES OF ITEMS EVALUATING 
PNSERUCTORS@ IMPRESSION, OF FIVE, ASRBECTS#OFeRiEewl DETPeTRAINER 





Question 


| How easily did IDCTT allow you to 
liibmstruct students? (Difficult..Easy) 


t How realistic was the IDCTT Trainer? 
PArtificial..Realistic) 


Extent you would like to see IDCTT 


installed permanently at the DCA School? 
(Very Little..Very Much) 


Rate the students reaction to the IDCTT 
Trainer? (Negative. .Positive) 


How beneficial would IDCTT be as a 
shipboard trainer? 
(Detrimental. .Beneficial) 


B. IDCTT VERSUS DC CENTRAL TRAINER PERFORMANCE COMPARISON 
tee same s2 students were used in the IDCTT versus DC Central Trainer 


performance comparison as those in the IDCTT Trainer performance 


evaluation. The data consisted of graded student performance and rating 
data. The results are presented below. 
Ge Graded Student Runs 


Each of the 32 student participants recelved two grades, one for 
their performance in the IDCTT Trainer and the other for the DC Central 
Trainer. These scores provided paired scoring data which was analyzed 
using the Wilcoxon Signed Rank Test. The Wilcoxon Signed Rank Test 
revealed that the population relative frequency distribution of the IDCTT 
Trainer performance scores was shifted to the right of the DC Central 
trainer performance scores (.025 < P € .01). Table 13 compares summary 
Statistics for the two sets of students’ scores, while Appendix C contains 


the raw data scores and the Wilcoxon Signed Rank Test results. 
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TABLE 13: SUMMARY STATISTICS FOR STUDENT SCORES RECEIVED DUR Seas 
PERFORMANCE EVALUATION SESSIONS FOR THE IDCTT TRAINER AND THE DC CENTRAL 
TRAINER 


Statistical Parameter IDCTT Trainer DC Central Trainer 
Parameter Parameter 
Mean 89.34 86.53 


| Standard Deviation 5.01 
-0.06 
T2 


v9 


Minimum Score Achieved 





Maximum Score Achieved 99 93 





2 Rating Data 

Scale values and interquartile ranges were compiled from data 
collected by the Student IDCTT vs DC Central Trainer Comparison Survey and 
the Scenario Topics Ranking Survey. Table 14 compares the scale values 
and interquartile ranges. Scale values span an 11 point bipolar scale: 
low scores indicate a preference for the IDCTT Trainer; high scores 
indicate a preference for the DC Central Trainer. a six 1S neuceome 

Table 15 shows the scale values, interquartile ranges and rank 
order for each fundamental damage control topic. These data were obtained 
by the IDCTT and DC Central Trainers’ Scenario Topics Ranking Survey. IE 


should be noted that unlike previous survey scales used in the present 
study which employed an eleven point range, the Scenario Topics Ranking 


Survey, students’ responded on a_seven point scale. The higher the scale 


value for a topic the more important the topic was in successfully 


completing the scenario problem. 
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TABLE 14: SCALE VALUES AND INTERQUARTILE RANGES OF ITEMS COMPARING 
= UDPEIwts) IMPRESSIONS OF THE IDCTT TRAINER AGAINST THE DC CENTRAL. TRAINER 
COMPARISON ACROSS 11 OPERATIONAL AND DESIGN CHARACTERISTICS 


Question Scale IQR 
Value 
Simulated the shipboard environment more 4.04 ee 
realistically 


Enabled instructors to provide complete 
post scenario debriefs 


Produced the greatest level of stress ZO 


Allowed instructors to monitor student’s pes xl pes 6 
performance 










Prepared the student for actual shipboard 
emergencies 
Updated student inputs more easily ote, Leto tf] 


Provided scenario information closely A250 A, 5:0 
resembling shipboard methods 


Provided most effective teaching 3-250 2.57 






environment to exercise damage control 
skills 


Promoted greater learning in time Zee 2-47 
allotted ; 


Preferred training method 1.39 2.89 


Stimulated the student to perform 
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TABLE 15: SCALE VALUES, INTERQUARTILE RANGES AND RANK ORDERS COMPARING THE 
IMPORTANCE OF FUNDAMENTAL DAMAGE CONTROL TOPICS IN COMPLETING THE ieee 
AND LUC aC ENTERAL TRA NER SZ SOT ah’. oie PROBLEM 


Measure Scale Value 


TDCi aie PDC DC EDC ES 1D. 
—- Central ae 


| Communications Sa seal 

| Inform chain of Spe Pia) goal’ 10 
i ee 
[Set Zebra i 
Restore Vital us TRS ae ow 0. 35 1, ysl 
Systems 
Manage DC 
1 a eee Oe 


Isolate (ee Laliecrt | 61 | 6.07 07 ee 04 lig | 4.27 


Manned and Ready ae 
pssstere riveete | 200 ee ee 

Manage Personnel iL eke: 209 ie o'6 2no9 

ee 

Locate | eee | ace! 60 Soa 60 ” re) Sie J ey 

Prioritize Seaewl Ses c.Ene 204 

damage 


Coordinate fire 
fighting teams 


Isolate 
explosive 
hazards 





Ws DATA ANALYSIS AND DISCUSSION 

Data collected from the validation test group allowed two 
circumscribed analyses: an analysis of the IDCTT Trainer performance alone 
and a comparative analysis of the IDCTT versus the DC Central Trainer. 
This chapter analyzes the results for these two evaluations. The first 
part addresses the IDCTT Trainer performance evaluation and reports the 
findings from the short essay, rating data, check-off problem features, 
and instructor evaluations. The second part contrasts the IDCTT Trainer's 
performance with the DC Central trainer, using the rating data and short 


essay responses reported in the last chapter. 


A. IDCTT TRAINER PERFORMANCE EVALUATION 
The IDCTT Trainer performance evaluation’s objective was to collect 

sufficient data to determine the effectiveness of the IDCTT Trainer as a 
student training aid and an instructor teaching tool. The focus of the 
present treatment is four-fold. First, to determine if users easily 
interacted within the IDCTT environment and to examine features of the 
IDCTT Trainer which might be improved. Second, the scenario itself was 
examined to determine its effectiveness as a simulated battle training 
problem. Third, the actual student workload using the IDCTT Trainer was 
evaluated to determine if the trainer provided an acceptable training 
environment. Finally, the instructors’ evaluation of the IDCTT Trainer 
was examined to establish a basis in experienced opinion regarding the 
system’s usefulness as a training tool. These four themes are presented 
in the following four sections: 

- System User Interface, 

escenario Critique, 

e Student Workload, and 


« Instructor Evaluation. 
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i eee System User Interface 
The Student IDCTT Trainer Survey and the User Interface 
Dimensions Survey, were used to determine students’ reaction to the IDCTT 
Trainer’s hardware and software features. These surveys provided short 
essay information, rating data, and check-off problem features as 


discussed in Chapter Two. The results are discussed below. 


a. Student IDCTT Trainer Survey Analysis 


Scale values calculated from the Student IDCTT Trainer Survey 


indicated that students liked the IDCTT Trainer package and felt it was an 
extremely useful training aid (see Figure 2). Students generally agreed 


(IOR = 1.47) that the system was easy to operate (S Sm. Moreover, 
respondents indicated a slightly stronger than neutral attitude that the 
information options presented by the Command and Control Console were 
adequate to successfully combat the scenario (S = 7.09); however, their 
responses to this question tended to vary more widely (I10OR J2aauGm 
Interestingly, the survey revealed that students generally accepted the 
touchscreen monitor as an easy method for inputing data into the system (S 
= 7.91), despite comments in the essays and features check-off section 
which identified the touchscreen monitor as an area where improvements 
should be made.?° Students’ responses on items concerning the touchscreen 
monitor’s performance revealed a substantial lack of consensus (IOR of 


AED): 


The trainer’s ratings of usefulness as_a DC School training 
aid or as_a potential shipboard training aid, were very high and 


consistent across students. The usefulness ratings were 9.61 in each 


case, with very low IOQRs of 1.23 and 1.55 respectively. 
b. Operating System 


Students were clearly favorably impressed with the system’s 


operational characteristics. Despite the high positive scale values 


7° These £indings will be disensseq lateue 
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Student IDCTT Trainer Survey 


System Operations Results 
Negative Positive 
Responses Responses 


How Easy the System is to Operate 
(Difficult..Easy) 


Relevant Information Options On the 
Touchscreen Monitor (None..All) 


How Easily Touchscreen Allowed 
Infoonation Input (Difficult. Easy) 


Usefulness as a DC School Training 
Aid (Worthless..Useful) 


Usefulness as a Shipboard Training 
Aid (Warthless..Useful) 


Figure 2: Scale Values and Interquartile Ranges of Items Evaluating 
Student Impressions of Five Design Aspects of the IDCTT Trainer 


associated with the touchscreen monitor’s operational characteristics, 
most of the problems identified with the system were rooted in using the 
touchscreen monitor to input information into the system. The Check-oOff 
Problem Features List was administered to identify areas in the IDCTT 
Trainer which needed improvement. Figure 3, graphically displays the 
percentage of students who identified each of the seven IDCTT Trainer 
features as areas where they encountered difficulties.*} The following 
discusses these findings as they relate to seven specific features: 

* Operating the touchscreen monitor, 

¢ Understanding audio reports, 


¢« Locating DC Plate information, 


we The seventh IDCTT Trainer feature was an "other" category, enabling 
students to identify problem areas not listed. 
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«+ Speed or volume of information presented, 
- DC Alarm Panel display, 
+ Firemain Panel and firepump operations, and 


¢ Other features not listed. 


Check-Off Problem Features 


Percentage of Student Responses 


8 


cs 
2 
8 
c 


Touchscreen Audio DC Plate Paco ODOC Panel Fire Pane! Other 





Figure 3: percent of Students Indicating Problems with Seven IDCTT 
Trainer Features 


(1) Touchscreen Monitor. Fifty-nine percent of the students 


indicated the experienced some difficult operatin the touchscreen 





monitor. More specifically, they identified difficulties inputing 
information into the touchscreen as the cause for missing key audio 
reports, and in some cases, losing control of the battle problem. The 
touchscreen’s primary problem was its low sensitivity to touch and its 
slow response time. Students often selected the correct button, but the 
touchscreen did not sense their input. Further, the actual physical 


distance between the buttons was narrow, causing students to accidentally 
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choose the wrong option. This was particularly frustrating when inputing 
fire and flooding boundaries. Ordering boundaries required pressing a 
sequence of up to 14 buttons, taking upwards of 30 seconds of problem 
clock time to complete. An accidental input during this sequence could 
seriously affect the student’s performance by causing them to focus their 
attention on repeating the 14 button process rather than the flow of 
information from the various sources. Most of the students opted to use 
the computer keyboard instead of the touchscreen’s numeric key pad. 
However, switching back and forth between the computer keyboard and the 
touchscreen monitor was difficult because only numeric inputs and the 
enter command could be initiated from the computer keyboard. 

Many of the students had never used a touchscreen 
monitor before this study, but they became markedly faster with practice. 
Still, comments in student essays revealed an underlying sentiment to 
either increase the sensitivity of the touchscreen or explore different 
methods for inputing information. These suggestions included: 

* Voice activation, 
- Using a mouse, 
¢ Making all action available on the computer keyboard, and 


¢ Designating an individual to input all orders into the touchscreen. 


(2) Volume of dJnformacion-. The next most frequently 
reported problem concerned the speed or volume of information presented. 
Thirty-eight percent of the students indicated they had problems with the 
Pace of information flow, but student essay responses did not provide 
information as to why: only five students specifically addressed this 
area. Although some students seemed to be overwhelmed with the speed and 
volume of information during their practice run, most appeared to be 
comfortable with the pace by the time they reached their graded session. 
Some students specifically identified the fast pace as _ instrumental in 
making the trainer scenario seem more realistic. Interestingly, one 


student suggested programming different difficulty levels, such as 


au 


"beginner", "intermediate", and "advanced", thus enabling slower students 


to practice at levels more conducive to their stage of skill development. 


(3) DC Plate, Firemain Alarm Panel and Pump Operations. 
Approximately the same percentage of respondents identified "locating DC 
Plate information" (21 percent), and the "Firemain Alarm Panel and pump 
operations" (19 percent), as problem areas during their evaluation trials. 
Students attributed their difficulties locating DC Plate infGmmaeamege ee 
their unfamiliarity with the DDG-51 class. Although this is neers 
shortcoming of the IDCTT Trainer, it indicates the need for more scenarios 
with different ship classes to support the diversity of fleet experience. 
While the IDCTT Trainer was not designed to instruct students on reading 
Dc Plates, its continued use will familiarize students with the DDG-51 
class ship generally and reading their DC Plates specifically. 

Respondents indicated the main problem with the Firemain 
Alarm Panel was the small display screen. The Firemain Panel is displayed 
on a @15> inch ~“cetassemoniteor - Since the display screen symbolically 
represents over thirty of the primary firemain valves and six firepumps, 
the size of the valves and pumps are small given the limited screen space. 
Students often overlooked critical indications on the Firemain Panel 
because they simply did not notice any changes in the compact display. 
Two solutions to this problem are readily apparent. 
- A larger Firemain Panel Screen would proportionally increase the size 
of each Firemain Panel feature without altering the present display, 


and 


°° Reducing the number of valves displayed to those critical to the 
scenario, thus allowing more room for those remaining. 


Adopting a larger screen is preferred because it enables the students to 


actually see a representation of the DDG-51 class Firemain Panel. 


(4) Audio Reports, DC Alarm Panel, and Features Not Listed. 
The areas which needed little or no improvements were, “other features not 
listed in the survey" (16 percent), audio reports (6 percent), and the DC 
alarm panel (no responses). Unfamiliarity with the system, particularly 


not knowing what actions the IDCTT Trainer did automatically and what 


SZ 


orders had to be manually inputed to accomplish specific tasks, summarize 
the responses for the features not listed in the survey. Students 
suggested using a demonstration tape to provide a step by step sequence 
through the various options.” 

For audio reports, respondents indicated that they 
became distracted by erroneous information in the background noise. 
Background noise is simply a compilation of actual reports delivered to 
the DCA over the Command and Control Console. It is played continuously 
on a separate audio track. Unfortunately, many of these reports are 
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erroneous,°’? depending on the student’s previous corrective actions to 


combat the damage. This problem could be easily rectified by recording 
background information that 1s relevant, but general in nature. This 
would provide an appropriate distraction, but would not give the DCA mis- 
PaLOormation - 

No student in the study identified any problems with the 
DC Alarm Panel. Although the DC Alarm Panel was displayed on a 15 inch 
color monitor similar to the Firemain Alarm Panel, there were no problems 
reported with the size of the graphic representation of the various 


alarms. 


32 Fach student attended a 90 minute lecture on the background and functions 
of the IDCTT Trainer. This lecture used overhead transparencies of the IDCTT 
Trainer’s screen displays rather than the actual system. Respondents felt the 
overhead transparencies did not sufficiently prepare the students for the actual 
trainer. 

Although there is a demonstration option on the IDCTT Touchscreen, this option 
simply repeats a complete scenario, darkening the buttons that would be pressed 
for the various actions taking place on the screen. There 1s no audio voice over 
in the demo to explain the actions that are being taken. Further, watching the 
present demo tape prior to system use would predisclose the battle problem 
because it is the same scenario that the student would use. 


33 Erroneous reports include, "High Temperature Alarm in the VLS Magazine”, 
“Loss of Firemain pressure", and “Complete loss of Chill Water, no radar arrays 
on line". Fach of these reports potentially cause the student to take unnecessary 
Borrective actions. 
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c. User Interface Dimensions 
The User Interface Dimensions Survey measured student 
responses across eight conventional interactive courseware dimensions. 
Student responses were clearly positive on each dimension; the scale 
values ranged from 8.32 to 9.92. Further, there was little variation in 
the™ student responses the TOR@ranged™ fmemm. 43aRtom eee. | Figure 4 


depicts these results graphically. 


User Interface Dimensions 
Questionnaire Results 


Ease of Use (Difficult..Easy) 


Navigation Through ICW 
(Difficult..Easy) 


Cognitive Load 
(Unmanageable..Manageable) 


Mappmeg (None. Powerful) 


Knowledge of Space Compatibility 
(Incompatible..Compatible) 


Information Presentation 
(Obtuse..Clear) 


Media Integration 
(Uncooardinated. .Coardinated) 
Overall Functionality 
(Dysfuctional. Highly Functional) 





Figure 4: Scale Values and Interquartile Ranges of Items Evaluating 
Student Impressions on Eight ICW Interface Dimensions 


2. Scenario Critique 
Figure 5 reveals that student reactions to the IDCTT Trainer’s 
scenario were highly positive. The validation group reported that the 


scenario was easy to understand (S = 8.14) and was extremely realistic (S 


34 See Table 6 for a complete summary of the scale values and interquartile 
ranges for each interface dimension. 
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=.) clea Be Responses on both aspects of the scenario showed little 
variation as reflected in the narrow IORs of 1.87 and 2.27, respectively. 
Students were neutral and in agreement on the pace of the scenario (S = 
eeou (TOR of 1:34). Finally, the validation group agreed that the 
Beener1Oo was Gifficul€ Co complete (S = 4.39; IOR = 1./2). The following 
four sections address various aspects of the scenario used and student 


meace1ons to 1t. 


Student IDCTT Trainer 


Difficulty of Scenario 
(Difficult..Easy) 


Was the Scenario Easy to Understand 
(Confusing..Clear) 


Was the Scenario’s Pace too Slow 
(Fast..Slow) 


Was the Scenario Realistic 
(Unrealistic..Realistic) 





Figure 5: Scale Values and Interquartile Ranges of Items Evaluating 
Student Impressions of Four Scenario Design Aspects 


a. Scenario Realism 
Student narratives cited audio and visual effects as the main 
reasons why the trainer was so successful in creating what they considered 
a realistic training scenario. Further, they revealed that the realism 
was heightened in that the printed damage control chits provided another 


information source they would receive if they were performing in an actual 


mae, 


shipboard battle damage situation. Moreover, they reported that the 
information flow closely tracked that which they would experience in an 


actual shipboard multi-hit scenario. 


b. Scenario Clarity 

Essay responses and narratives, clearly indicated that the 
scenario objectives were clearly defined. Student responses repeatedly 
suggested that the progression of steps needed to complete the scenario 
tracked the standard procedures taught at the Damage Control School. 
Two items, however, were identified as potential problems with 
understanding the IDCTT Trainer scenario. The first was an insufficient 
knowledge of the chill water system to make rational decisions on the 
corrective actions necessary to isolate and repair it. The second problem 
stemmed from an underlying unfamiliarity with the DDG-51 Class Destroyer 
which made it difficult for students to fully comprehend the information 


they received. These two topics are amplified. 


(1) Chill Water. Some of the students identified ignorance 
OF meni ieee system as the basis for their inability to successfully 
complete the scenario. Fourteen of the 32 students (44 percent) did not 
complete the scenario due to chill water related problems. The IDCTT 
Trainer scenario emphasized chill water system restoration as one of the 
primary actions needed to complete the problem, and did so to attain the 
central goal of incorporating TSS concepts into the battle problem. The 
chill water system, however, was not taught at the Damage Control School 
and students were left with the limited knowledge they obtained at 
previous schools or experienced in the fleet. The Damage Control School 
should review its course curriculum and place a stronger emphasis on the 
Chill water system if the curriculum is to reflect the shift in fleet 


doctrine toward the TSS philosophy. 


73> Successful completion of the scenario requires students to investigate 


damage, set boundaries, combat damage, and repair the damage. This broad 
sequence of events are the fundamental steps taught to damage control students 
in managing damage control situations. 
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(2) DDG-51 Class Unfamiliarity. Another scenario related 
problem was that the scenario was based on a class of ship with which most 
were unfamiliar. Since the DDG-51 Class is the newest ship class, most of 
the students were unfamiliar with its design, making it difficult to 
locate compartments and valve numbers from the DC Plates. Although the 
students were not familiar with the DDG-51 Class design, the concepts used 
to combat damage are the same, regardless of the ship’s design, and 
students who employed solid damage control concepts were able to 
successfully complete the scenario, even if they did not know the ship 
class specifics. When scenarios are written for different ship classes, 
students will be able to choose their ship class from a library of 
scenarios, tailoring training to the knowledge they need to bring back to 


the fleet. 


c. Scenario Pace 
Students’ reaction to the scenario pace was neutral. 
Although some students fell behind in the problem, they attributed the lag 
to the slow process of inputing information through the touchscreen rather 
than the flow of information itself. Most students felt that the fast 
flow of information added to the realism of the scenario and induced a 
level of stress that would be present under an actual damage control 


scenario. 


d. Scenario Difficulty Level 

Students felt the scenario was difficult and only one out of 
32 students successfully completed the scenario during the practice 
session. This low completion was partially attributed to the students’ 
unfamiliarity with the system’s operating procedures during the practice 
run. However, the low completion rate underscores the difficulty of the 
scenario. The scenario was not designed to be easy, but to provide a 
challenging learning environment in which students could practice the 
damage control concepts they learned in the classroom. Further, it was 


designed to identify student mistakes so this knowledge could be applied 
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and those mistakes rectified in subsequent trainer sessions. Thus, the 
students’ prevailing experiences that the scenario was difficult reflected 


the designers original intentions: the exercise was to be difficult. 


a Student Workload 

Student workload was evaluated using NASA's Source of Workload 
Evaluation. The IDCTT Trainer was developed to produce a eee 
environment that imposed significant mental, temporal, and attentional 
demands.’® Respondents rated the system as mentally and temporally 
demanding while requiring a high level of effort to complete the IDCTT 
Trainer scenario. Scale values for these categories ranged from 9.21 to 
9.76, with very narrow ranges (IQRs of 1.79 and 1.90, respectively). As 
one would expect from a computer based simulation, students rated the 
physical demand required by the system as low (S = 3.50). When asked to 
rate the level of frustration they encountered while interacting with the 
system, student responses were essentially neutral (S = 6.50), but 
reflected significantly more variation in that assessment (IOR ="45759% 

A review of students’ performance grades indicated that those who 
performed poorly were more likely to be frustrated while using the system 
than those who performed well or vice versa. This finding explains the 
wide range of responses to the "level of frustration" survey question. 
Most students were satisfied with their personal performance level (S = 
8.86; IQR = 2.36). Figure 6 graphically depicts the student responses for 


the various categories. 


4. Instructor Evaluation 
The instructors’ evaluations of the IDCTT Trainer were generally 
neutral. Figure 7 shows the instructors’ responses. Unfortunately, only 
three of the seven instructors at the DCA school (43 percent) spent more 


than two hours operating the system and assisting students. The remaining 


76 See the Subject Instructions and Survey for the Sources Of (Wore 


Evaluation contained in Appendix B for a complete description of each workload 
faccor: 
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Source-of-Workload Evaluatiom 
Results 


Mental Demand a eo 
Physical Demand ————— 

Temporal Demand +--+ 
Performance Demand -—_-x 
Effort Demand os 
Frustration a aroma 


Figure 6: Scale Values and Interquartile Ranges of Items Evaluating 
Student Impressions of Six Human Workload Demand Dimensions 


jee bhe 


instructors formed their opinion on the system by: 
Personally completing two scenario runs, and 


Observing the other instructors assisting students during their 
practice and test sessions. 


Geula account for their neutral sentiments for: 
How easily the IDCTT allowed them to instruct students, 
How realistic the IDCTT Trainer was, and 


The extent they would have liked to see the IDCTT Trainer permanently 
installed at the DCA School. 


Instructor interviews revealed that the reason for the neutral response 


concerning the permanent installation of the IDCTT Trainer at the DCA 


School was that a clear mission in the school’s curriculum was not clearly 


defined. Since the system was not formally programmed to replace the DC 


Central Trainer, some instructors were not supportive of a separate IDCTT 


oe, 


Trainer requirement. They did acknowledge, however, the students’ 


positive acceptance of the system (S = 7.50; IOR = 1.50). 


IDCTT Trainer Instructor 
Survey Results 


How Easily Did IDCTT Allow You 
to Instruct Students (Difficult. Easy) 


Extent You Would Like to See 
IDCTT Installed Permanently at the 
DCA School 

(Very Little..Very Much) 


Rate the Students’ Reaction to the 
IDCTT Trainer (Negative..Positive) 


How Beneficial Would IDCTT Be as 
a Shipboard Trainer 
(Detrimental. Beneficial) 





Figure 7: Scale Values and Intergquartile Ranges of Items Evaluating 
Instructor Impressions of Five IDCTT Trainer Design Features 


B. IDCTT VERSUS DC CENTRAL TRAINER PERFORMANCE COMPARISON 

The DC Central Trainer provided a clear-cut baseline against which to 
compare the IDCTT Trainer. Since students were exposed to each system in 
a 24 hour period, they responded to the IDCTT Trainer vs DC Central 
Comparison Survey with recent experience in using each system. Further, 
ranking the importance of various damage control fundamentals using the 
Scenario Topics Ranking Survey, provided a measure for assessing the 
differing importance they attributed to 13 fundamental damage control 
topics for each trainer. Once the level of importance for each of these 
topics was established, the topics’ relative importance for each of the 


two trainers was compared to determine which method provided a more 


standardized scenario. Finally, the performance grades provided a simple 
way to evaluate student effectiveness in using the two trainers to combat 


the same battle problem. These comparisons are discussed below. 


: Student Trainer Comparison 
Results from each category on the Comparison Survey clearly 
indicated that students preferred the IDCTT Trainer over the present DC 
Central Trainer. The most emphatic response from this comparative survey 
was the computed scale value of 1.39 which indicated that the IDcCTT 
Trainer is, by far, the preferred training method.*’ Respondents also 
strongly indicated that the IDCTT Trainer promoted greater learning, 
produced significantly more stress, and stimulated them to learn much more 
mean Ehe DC Central Trainer did (S = 2.38, 2.50, and 2.38; IOR = 2.47, 
3.26, and 2.47, respectively). Many students described a greater sense of 
accomplishment after completing the IDCTT Trainer, since it seemed more 
emalienging. Although the instructors felt that the IDCTT Trainer did not 
Significantly increase their ability to evaluate student performance, the 
students themselves strongly felt that the IDCTT Trainer provided the 
instructor with a better method to assess their problem solving 
Membaormance (S = 2.71; IOR = 2.50). 
Responses on three of the comparison survey questions yielded no 
consensus among student responses. These questions were: 


* Method which better prepares students for shipboard casualties 
ee 24-06; IOR of 4.29), 


mebaster method to initiate actions (S = 3.83; IOR of 6.00), and 


¢ Method with the most realistic information presentation (S = 4.50; 
mer or 4 050') . 


These three items, and their attendant variability, appeared to be based 
two schools of thought on how actions should be initiated, and information 
transferred in simulated battle problem training. One group of students 


preferred the Command Console’s simulated shipboard environment and push 


37 Low values indicate a preference for the IDCTT Trainer while High values 
indicate a preference for the DC Central Trainer. 


buiston adaisplayss= The other group felt that the interaction among 
watchstanders was a key portion of the training experience and preferred 
using numerous Sound Powered Phone Talkers to relay and receive 
information. The critical design feature upon which these responses 
depended, however, was how much difficulty students had when they used the 
touchscreen monitor. Those who experienced difficulties using the 
touchscreen, preferred the DC Central Trainer method of initiating actions 
through Sound Powered Phone Talkers. Students who did not experience 
problems with the touchscreen, preferred the IDCTT Trainer's method. 
Preferences were clearly related to the users mastery of the touchscreen 
features. Figure 8 graphically displays the student responses across the 


11 comparative dimensions. 


Student IDCTT vs DC Central 
Trainer Comparison Survey 
Results 


Realistic Environment 
Post Scenario Debriefs 
Greatest Level of Stress 
Instructor Monitaring 


Prepares Student for Actual 
Casualties 


Initiate Damage Control Actions 
Realistic Information Presentation 
Mare Effective Teaching Medium 
Promotes Greater Leaming 
Preferred Training Method 
Stimulated You to Perform 





Figure 8: Scale Values and Interquartile Ranges of Items Comparing 
Student Impressions of 11 Trainer Design Features 
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as Student Performance Grades 
Student performance grades, collected from the two training 
methods, were used to determine which method provided a better environment 
in which students performed. As discussed previously, the scenario 
difficulty level was judged to be the same and students were graded using 
ee came grading criteria for each method.” The Wilcoxon Signed-Rank 
Test for a Paired Experiment was used to determine if the relative 
frequency distribution of student performance scores were identical, orif 
the IDCTT Trainer scores were higher than the DC Central Trainer scores. 
The results of this test identified which medium promoted higher student 
performance. Table 16 summarizes the results of this test while a 
complete set of calculations is included in Appendix C. 
TABLE 16: WILCOXON SIGNED-RANK TEST SUMMARY FOR STUDENT PERFORMANCE 
S€OrES FROM THE IDCTT AND DC CENTRAL TRAINERS 
ee a army Wm re 

Null Hypothesis: H,: The population distributions for the IDCTT 
Trainer and the DC Central Trainer performance scores are 
identical. 

Alternative Hypothesis: H,: The population relative frequency 
distribution of the IDCTT Trainer’s performance scores is 
shifted to the right of the DC Central Trainer’s performance 
scores. 


Test Statistic: Z = AOS 


Rejection Region: Reject H, 1f z 2 Z, 
where z, = 1.96 at the .025 significance level 


Conclusion: Reject H, and accept H, at the .025 significance level. 


TT aes, | a mm 
The Wilcoxon Signed-Rank Test indicated that the frequency 
distribution of scores received using the IDCTT Trainer were higher than 
those scores received using the DC Central Trainer (.025 < P< .01). The 
higher scores obtained from the IDCTT Trainer can be attributed at least 


three factors which are discussed below. 


48 Appendix A provides a copy of the student grading criteria used in the 
ScLucy . 


a. Student Preference for the IDCTT Trainer 

That students preferred the IDCTT Trainer to the DC Central 
Trainer was clearly reflected by the scale values from the Comparison 
Survey. In particular, they held that the IDCTT Trainer promoted greater 
learning, and stimulated them to perform to the best of their ability. 
Higher test scores, therefore, could be accounted for by students who were 
simply more motivated to perform in the IDCTT Trainer. In thea 
narratives and verbal remarks after the evaluation period, students 
indicated that the clear flow of information associated with the IDCTT 
Trainer Was critical to thelr geuccesee Conversely, in the DC Central 
Trainer, students reported that they were dependent on the abilities of 
their Phone Talkers to relay the correct information, a dependency which 


sometimes hindered the timely flow of accurate information. 


b. System Stress 

The IDCTT Trainer was designed to induce significant amounts 
of stress on students through the rapid pace of the scenario and the 
content of various audio and visual stimuli. The subject of performance 
under stress is not the focus of this paper, but it was readily observed 
that students who were exposed to the stressful IDCTT scenario appeared to 
concentrate more intently, and try harder to successfully complete the 
battle problem than when operating in the less stressful DC Central 
Trainer environment. When in the less stressful DC Central Trainer, 
participants projected a lax attitude toward completing the battle 
problem, and in general, did not seem to take the scenario as seriously as 
they did while executing the IDCTT scenario. The degree to which students 
became engrossed in the scenario seemed to correspond with their 


performance scores. 


c. Grading Expectations 
Before the test period, performance scores were expected to 
be higher in the DC Central Trainer due to students’ familiarity with the 


system and the concerted problem solving methods the group trainer 
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allowed. Two tactors “discounted this expectation. First, students 
quickly became familiar with the IDCTT Trainer. Most students eT il 
operated the IDCTT Trainer by the end of the practice session, and by the 
graded session, each student completed all required actions without 
outside assistance. The speed with which students adapted to the computer 
simulation quickly eroded any advantage the DC Central Trainer conferred 
because of its familiarity. Second, when students were removed from the 
group atmosphere of the DC Central Trainer, they assumed greater 
responsibility in solving the problem and decision making became more 
spontaneous. ine “Guality “Of decisions did mot - Significantly ditter 
between the two systems; per se, however, the speed with which decisions 
were made was much quicker in the IDCTT Trainer. Thus, the DCA’s 


performance did not seem to be hindered by not having a watchteam to 


suggest various damage control actions. 


d. Summary 

Based on student performance, the IDCTT Trainer is a more 
effective training medium. Students were more motivated to perform, and 
preferred the methods used by the IDCTT Trainer to deliver simulated 
battle problems. The IDCTT Trainer allowed the instructor to carefully 
monitor student performance by eliminating the need for the instructor to 
spend much of his time initiating the scenario rather than critiquing it. 
Unlike the DC Central Trainer, where the instructor is removed from the 
student training area, the IDCTT instructor 1S positioned to critique and 


assist the student in the training area. 


a. Trainer Type Scenario Variation 
Developing a procedure to measure the level of variation induced 
by each method over many repetitions of the same scenario was difficult. 
The objective of this portion of the study was to measure the extent to 
which each trainer consistently delivered a standardized scenario. 
Clearly, reducing the level of variation between repetitions so each 


student is given the same difficulty level in which to train is desirable. 
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Likewise, reducing the subjectivity in assigning performance scores 
between training periods with varying levels of difficulty is desirable 
from a methological perspective. That students would feel the computer 
simulation would provide more consistent training scenarios was expected 
because the computer generated scenario did not rely exclusively on the 
instructor’s ability to present the scenario, as is the case with the DC 
Central Trainer. The IDCTT Trainer is physically programed to execute a 
series “of “events which presents the same scenario™ cach umm =| 
Alternately, the DC Central Trainer’s scenario, which relies on the 
instructor to create the training environment, differs between trials for 
four important reasons: 


¢ Instructors do not follow a specific time-line when initiating the 
scenario, 


¢ Instructors are given a script of events but are free to execute each 
event in his own training style, 


- Instructors often adjust the scenario difficulty level to meet 
student performance level, and 


¢- Students can request additional information from the instructors, 
whieh 1s Net am Opticn wlth sthe TUeitraiereeal tet - 


The measures used to analyze the level of variation present in each 
trainer type were the scale values and interquartile ranges calculated 
from responses on the Scenario Topics Ranking Survey. These two measures 


are more fully discussed below. 


a. Scale Value Assessment 
The numerical index used to evaluate the variability in the 
two methods reflected the degree each method emphasized the various 
fundamental damage control actions required to complete the scenario. The 
Scenario Topics Ranking Survey required students to rate how important 
each of 13 fundamental damage control topics was in completing the 
scenario. The topics for each of the two systems were assigned ranks 


based on scale values obtained by the Scenario Topics Ranking Survey. 


79 The scenario does differ depending on the actions taken by the student- 
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Since the scenarios were the same for the two trainers, the scale values 
assigned for each topic should have been approximately the same. 

Simply stated, the ranks for the two systems tracked closely, 
except for communications, which was ranked the eleventh most important 
for the IDCTT Trainer and the third most important for the DC Central 
Trainer. Scale values for each topic fell within one point for each of 
the topic between the two methods with the exception of: 

¢+ Communications, and 

¢ Keeping the chain of command informed, 
which differed by only 2.05 and 1.14 points respectively. Figure 9 
displays the scale values obtained for the two trainer types. The figure 
highlights the more pronounced emphasis, as determined by the scale 
values, for the topics using the IDCTT Trainer. These scale values 
indicate that students felt the degree with which each topic was 


emphasized was more pronounced using the IDCTT Trainer. 


(1) Ranking Differences. As shown in Figure 9, 
communications and keeping the chain of command informed, are given 
different priorities in the two systems. The content of student 
narratives revealed that communications were not emphasized in the IDCTT 
Trainer because they did not feel the touchscreen input device effectively 
tested, or even required, their communication skills. Students felt the 
DC Central Trainer’s group environment, combined with use of Sound Powered 
Phone Talkers to exchange information, emphasized the need for well 
practiced communication skills. 

Students identified problems exchanging information with 
the Bridge and the Commanding Officer as a communication deficiency in the 
IDCTT Trainer. This shortcoming may have induced students to feel that 
keeping the chain of command informed was not a high priority in the IDCTT 
Trainer. Conversely, the instructor in the DC Central Trainer, acting as 
the Bridge and the Commanding Officer, continually prompted the student 


for information, thus creating a distinct need to keep the chain of 
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Scenario Topics Rankings 


Survey Results 
Not At All Extremely 
Important Important 
lL Zz 3 4 5 6 7 
Communications 7 
= e “. a ~~ DC Central 
Inform Chain of Cammand Be if Trainer 
Set Zebra Ne 
= 
Restore Vital Systems S 
Manage DC Central i 
Isolate Damage _—_ 
Manned and Ready Reports aaa 
Restore Firemain E 
Manage Personnel Casualties =< 
Locate Damage | 
Prioritize Damage A\ 
Coordinate Firefighting Teams ‘> 
| 
Isolate Explosive Hazards : 


Figure 9: Fundamental Damage Control Topics Scale Values for the IDCTT 
and DC Central Trainers 


command informed. Although the data did not indicate which method was 
more variable, it did demonstrate that each method emphasized the same 


damage control fundamentals with the exception of communications. 


b. Interquartile Range Assessment 

Overall, response variability for the IDCTT Trainer was 
higher than the DC Central Trainer. As shown in Figure 10, nine out of 
the 13 topics (69 percent) had a higher IOR value for the IDCTT Trainer, 
Students’ assessment of how important each topic was in completing the 
scenario varied more for the IDCTT Trainer. If, in fact, grading the 13 
damage control topics is a valid measure of effectiveness for evaluating 
system variability, this would indicate that the scenario produced using 
the DC Central Trainer is less variable than the IDCTT Trainer. This 


finding is contrary to the logical expectation that the computer 
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simulation 1s more suitable, by design, for providing identical training 
scenarios. If the IDCTT Trainer actually provides more consistent 
scenarios, then either the measures of effectiveness for evaluating system 
variation are incorrect or the testing methods for soliciting students 
responses were unclear or not the best method possible. Further data and 
analysis of the systems’ variation in producing consistent training 
scenarios 1s required. 
Scenario Topics Rankings 
Survey IQR Results 


Low 
IQR 


Communications 

Inform Chain of Command 
Set Zebra 

Restare Vital Systems 
Manage DC Central 

Isolate Damage 

Manned and Ready Reports 
Restore Firemain 

Manage Personnel Casualties 
Locate Damage 

Prioritize Damage 
Coordinate Firefighting Teams 


Isolate Explosive Hazards 





Figure 10: Fundamental Damage Control Topics Interquartile Ranges for 
the IDCTT and DC Central Trainers 


Cc. Summary 
The data collection plan provided sufficient information to conduct 
the validation analysis. An analysis of collected data indicated that the 


IDCTT Trainer is an effective training tool, however there are areas where 
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improvements can be made. The following chapter summarizes the findings 


of this validation study. 


70 


V. CONCLUSIONS 

The findings of this validation indicate that the IDCTT Trainer is a 
highly effective training aid in a shore based environment. Students not 
only reported that felt the IDCTT Trainer was easy to operate and 
extremely useful as a training aid at the Damage Control School, they 
indicated they would like to see the trainer made available for shipboard 
use. When compared to the DC Central Trainer on specific design features, 
respondents indicated a clear preference for the IDCTT Trainer on every 
dimension examined. Further, the distribution of eed performance 
scores taken from the two different trainers revealed that the IDCTT 
Trainer is clearly the more effective training medium. The following 
sections encapsulate this study’s major findings from both the IDCTT 
Trainer performance evaluation and the IDCTT versus DC Central Trainer 


performance comparison. 


A. IDCTT TRAINER PERFORMANCE EVALUATION 

Taken together, the data from the IDCTT Trainer performance evaluation 
revealed that the system was enthusiastically accepted by the student test 
group. While instructors themselves were neutral in their impression of 
the IDCTT Trainer, they indicated that their students reacted positively 
to the system. The results from this circumscribed portion of the study 
were presented in two broad categories; IDCTT Trainer performance 
attributes, and suggested areas of improvement. The findings from these 


topics are summarized. 


ale IDCTT Trainer Performance Attributes 
Students indicated that the IDCTT Trainer was an extremely 
effective training aid and the IDCTT Trainer performance evaluation 
specified the following desirable attributes. 
¢ Students liked the IDCTT Trainer concept and design. 


¢ The IDCTT Trainer was easy to operate. 


ral 


The trainer was consistently considered useful as a DC School 
training aid or as a potential shipboard training aid across all 
students surveyed. 


Student responses were clearly favorable to the IDCTT on each of the 
eight conventional interactive courseware dimension presented. 


The scenario was easy to understand and was extremely realistic. 
Audio and visual effects created by the trainer were identified as 
the main reasons why the trainer was so successful in creating what 
the students considered to be a realistic training scenario. 

The students’ prevailing experiences that the scenario was difficult 
reflected the designers original intentions: the exercise was to be 
Git piculee 

Scenario objectives were clearly defined. 

Respondents rated the system as mentally and temporally demanding 
while requiring a high level of effort™to complete the IDCT@3@2a07-. 


scenario. 


Instructors acknowledged the students’ positive acceptance of the 
system. 


Students quickly became familiar with the IDCTT Trainer. 


Bee Improvement Items 


Although the IDCTT Trainer was agreed upon by the students to be 


an effective training medium, various characteristics of the trainer were 


identified for potential improvements. These areas identified by the 


validation study are listed below. 


Most of the problems identified with the system were rooted in 
inputing information into the system using the touchscreen monitor. 


Fifty-nine percent of the students indicated they experienced some 
difficulty operating the touchscreen monitor. 


The touchscreen’s primary problem was its low sensitivity to touch 
and its slow response time. 


An accidental input into the touchscreen monitor could seriously 
affect the student’s performance by causing them to focus their 
attention on repeating the button process rather than the flow of 
information from the various sources. 


Students recommended increasing the sensitivity of the touchscreen or 
exploring different methods for inputing information into the system 
such as voice activation. 


Thirty-eight percent of the students indicated they had problems with 
the pace of information flow. However, most students appeared to be 
comfortable with the pace by the time they reached their graded 
session. 


Le 


Students and Instructors indicated the need for more scenarios with 
different ship classes to support the diversity of fleet experience. 


The main problem identified with the Firemain Panel was the small 
display screen. 


Students often overlooked critical indications on the Firemain Panel 
because they simply did not notice any changes in its compact 
display . 


Students suggested a demonstration tape to provide a step by step 
sequence through the various options in order for the students to 
familiarize themselves with the system more quickly. 


For audio reports, respondents indicated that they became distracted 
by erroneous information in the background audio track. 


Fourteen of the 32 students did not complete the scenario due to 
chill water related problems. 


The Damage Control School should review its course curriculum and 
place a stronger emphasis on the chill water system if the curriculum 
is to reflect the shift in fleet doctrine toward the TSS philosophy. 


Some instructors were concerned that the IDCTT Trainer lacked a clear 
mresion in the school’s: curriculum. 


IDCTT VERSUS DC CENTRAL TRAINER PERFORMANCE COMPARISON 


The IDCTT versus DC Central Trainer performance comparison was used 


to measure the IDCTT Trainer’s performance against the DC Central Trainer. 


This portion of the study revealed that students felt that the IDCTT 


Trainer was the preferred training method in every category examined. 


Specifically, the following keynotes were identified. 


The IDCTT Trainer is by far the preferred training method. 


The IDCTT Trainer promoted greater learning, produced significantly 
more stress, and stimulated Students to learn much more than the DC 
Central Trainer did. 


The frequency distribution of scores received using the IDCTT Trainer 
were higher than those scores received using the DC Central Trainer 
at the .025 significance level. 


Based on student performance, the IDCTT Trainer is a more effective 
training medium. - 


When students were exposed to the stressful IDCTT scenario, they 
appeared to concentrate more intently, and try harder than during the 
DC Central Trainer sessions. 


Students assumed greater responsibility in solving the scenario and 


decision making became more spontaneous while exercising in the IDCTT 
training environment. 


is 


Students’ assessment of how important each fundamental damage control 
topic was in completing the scenario varied more for the IDCTT 
Trainer, indicating less standardized scenarios than the DC Central 
Trainer. 


Instructors felt that the IDCTT Trainer better prepared the DCA 


students for the actual damage control situations they will face in 
the fleet. 
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APPENDIX A SURVEYS 


The following surveys were the surveys used in support of the data 
collection plan. All seven surveys and grade sheets which were used in 


the study are included in this Appendix. 


STUDENT IDCTT SURVEY 


NAME DATE 

RANK YEARS OF SERVICE 

YEARS ENLISTED SERVICE PRIOR ENLISTED RATE 

WHAT SHIPS HAVE YOU SERVED ON? HOW LONG 


Have you served as a Damage Control Assistant for 6 months or more (CIRCLE 
ONE)? Yes No 


Have you served as repair 2, 3 OR 5 Locker Officer for 6 months or more 
(CIRCLE ONE)? Yes No 


Your answers to the following questions will help improve the quality 
of training you receive in the Damage Control Central Simulators. Please 
answer the following questions completely, explaining your answers 
thoroughly. Use the Back of the questionnaire if additional answer space 
ls required. Upon completion, please return this survey to your 
INSeeuCctor., 


1. Approximately how much time did you spend using IDCTT? 
hours 
2. Have you used interactive video courseware such as the IDCTT system 
before? (CIRCLE ONE) Yes No 
If yes, what courseware did you use? (use back if necessary) 


3. Rate how difficult or easy the IDCTT System was to operate? 


Very Very — 
DIFELZEOLE Neutral Easy 
1 i 2 4 5 6 7 8 y) 10 Adak 


4. Check any of the following operations which caused you difficulty while 
operating the IDCTT System. 


Inputing information with the touchscreen monitor 


BS 


Understanding audio reports 


Finding DC plate information with ISMS (examples: compartment and 
valve numbers) 


Speed or volume of information presented (did you easily lose track 
of the situation due to the speed or volume of information flow) 


Damage control alarm panel display 
Firemain panel and firemain valve and pump options 
Other (please specify): 


5S. In the space provided below, briefly explain why the items you checked 
eemeed you aifficulty. 


6. Rate the extent to which the touchscreen allowed you to input the 
information necessary to combat the Damage Control scenario. 


All of the None of the 
Information Neutral information 
il 2 3 4 5 6 7 8 9 10 agi 


7. Rate how easily the touchscreen allowed you to input information 


Very Very 
MrELicult Neutral Easily 
1 2 3 4 . 6 a 8 9 10 dBi 


8. How can the touchscreen control panel be improved? 


9. Rate the IDCTT Scenario according to the following criteria (note: this 
question refers to the battle problem itself and not the IDCTT system as 
a whole). 


TOO TOO 
Easy Neutral DIEFICule 
1 Z 3 4 5 6 it 8 9 10 joa 
Very Very Easy 
Confusing Neutral to Understand 
AL 2 3 4 5 6 7 8 9 10 Na 
Too Too 
Fast Neutral Slow 
1 2 : 4 5 6 Z 8 9 1) al: 
Very Very 
Realistic Neutral Unrealistic 


1 2 e) + S 6 q 8 9 10 ls 


~] 
~] 


10. What problems did you encounter while using the IDCTT trainer? 


11. What aspects of the IDCTT Trainer do you like the most? 


12. What aspects of the IDCTT Trainer do you like the least? 


13. Rate how useful the IDCTT Trainer is as simulation training aid for 
students at the Damage Control School? 


Very Completely 
Useful Neutral Worthless 
dL Z 3 4 5 6 de ae 9 10 Ee 


14. Rate how beneficial the IDCTT Trainer would be as an installed 
shipboard training aid. 


Very Completely 
Beneficial Neutral Detrimental 
il 2 3 4 5 6 7 8 9 10 fla 
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SUBJECT INSTRUCTIONS AND SURVEY: 
SOURCES-OF WORKLOAD EVALUATION 


Throughout this experiment the rating scales are used to assess you 
experiences in the different tasks conditions. Scales of this sort are 
extremely useful, but their utility suffers from the tendency people have 
to interpret them in individual ways. For example, some people feel that 
mental or temporal demands are the essential aspects of workload 
regardless of the effort they expended or the performance they achieved. 
Others fell that if they performed well the workload must have been low 
and vice versa. Yet others feel that effort or feelings of frustration 
are the most important factors in workload; and so on. The results of 
previous studies have already found every conceivable pattern of values. 
In addition, the factors that create levels of workload differs depending 
on the tasks. For example, some tasks might be difficult because the must 
be completed very quickly. Others may seem easy or hard because of the 
intensity or mental or physical effort required. Yet others feel 
difficult because they cannot perform well, no matter how much effort is 
expended. 


The evaluation you are about to perform 1s a technique that has been 
developed by NASA to assess the relative importance of six factors in 
determining how much workload you experienced. The procedure is simple: 
Read the following task descriptions and then mark the scale at the point 
that reflects the task load that you experienced. If you have any 
questions, please ask them now. Thank you for you participation. 


Pitle Endpoints Descriptions 


Mental Demand Low/High How much mental and perceptual activity was 
required (e.g., thinking, deciding, 
calculating, remembering, looking searching, 
etc.)? Was the task easy or demanding, simple 
or complex, exacting or forgiving? 


Physical Demand Low/High How much physical activity was required (e.g., 
Dueling pulling, turmane,. controlling, 
activating, etc.)? Was the task easy or 

demanding, slow or brisk, slack or strenuous, 
restful or laborious 


Temporal Demand Low/High How much time pressure did you feel due to the 
rate or pace at which the tasks or task 
elements occurred? Was the pace slow and 
leisurely or rapid and frantic? 


Beet Low/High How hard did you have to work (mentally and 
physically) to accomplish your level of 
performance? 

Performance Low/High How successful do you think you were in 


accomplishing the goals of the task set by the 
experimenter (or yourself)? How satisfied were 
you with your performance in accomplishing 
these goals? 


Pmeustration Low/High How insecure, discouraged, irritated, stressed 
and annoyed versus secure, gratified, content, 
relaxed and complacent did you feel during the 
task? 
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MENTAL DEMAND 

How much mental and perceptual activity was required (e.g., thinking, 
deciding, calculating, remembering, looking searching, etc.)? Was the 
task easy or demanding, simple or complex. exacting or forgiving? 


Low High 
1 Y, 3 4 5 6 7 8 S 10 tel 
PHYSICAL DEMAND 
How much physical activity was required (e.g., pushing, pulling; tune 
controlling, activating, etc.)? Was the task easy or demanding, Slowsor 
brisk, slack or strenuous, restful or laberious- 
Low High 
il 2 5 4 5 6 e 8 9 10 Ail 
TEMPORAL DEMAND 
How much time pressure did you feel due to the rate or pace at which the 
tasks or task elements occurred? Was the pace slow and leisurely or rapid 
and frantic? 
Low High 
1 2 3 4 5 6 a 8 9 10 ee 
PERFORMANCE DEMAND 
How successful do you think you were in accomplishing the goals of the 
task set by the experimenter (or yourself)? How satisfied were you with 
your performance in accomplishing these goals? 
Low High 
i} Z ) 4 5 6 7 8 9 10 Hi 
EFFORT DEMAND 


How hard did you have to work (mentally and physically) to accomplish your 
level of performance? 


Low High 
al 2 | 4 5 6 7 8 9 10 sl 
FRUSTRATION 


How insecure, discouraged, irritated, stressed and annoyed versus secure, 
gratified, content, relaxed and complacent did you feel during the task? 


Low High 
1 2 3 4 5 6 7 8 9 10 11 
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USER INTERFACE DIMENSIONS 


Directions: A number of statements which describe the Interactive-video 
Courseware (ICW) are given below. Read each statement and then make 
circle the number which reflects your opinion. There are no right or 
wrong answers. 


Dimension 1 - Ease of Use 
(perceived facility with which user interacts with the ICcwW) 


DrtelErecult Easy 
1 2 3 4 5 6 a 8 9 10 a 
Dimension 2 - Navigation 


(perceived ability to move through the contents of the ICW) 


PEEeLoulte Easy 
i Z 3 4 5 6 a 8 9 0 LA 
Dimension 3 - Coqnitive Load 


(Perceived degree that the user interface seems manageable) 


Unmanageable Manageable 
iN fi : 4 =) 6 7 8 9 10 ales 


Dimension 4 - Mapping 
(Program’s ability to track and graphically represent user’s path through 


the program) 


None Powerful 
iL 2 3 4 5 6 7 8 9 10 ml 


Dimension 5 - Knowledge Space Compatibility 


(Network of concepts and relationships that compose the user’s knowledge 
about the topic) 


Incompatible Compatible 
HI 2 3 4 5 6 f 8 9 16 deal 
Dimension 6 - Information Presentation 


(Perceived degree that the information contained in the ICW is presented 
in an understandable form) 


Obtuse Clear 
iL 2 3 4 5 6 oO 8 9 10 (mal 


Dimension 7 - Media Integration 


(How much does the ICW coordinate the different media to produce an 
effective whole) 


Uncoordinated Coordinated 
1 Z 3 4 5 6 7. 8 g 106 An 


ce 


Dimension 8 - Overall Functionality 


(Perceived utility of the ICW in relation to the program’s intended use) 


Dysfunctional Highly functional 
fi 2 3 4 5 6 a 8 9 10 dig 
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IDCTT INSTRUCTOR SURVEY 


NAME DATE 
RANK YEARS OF SERVICE 


1euIS AS AN INSTRUCTOR AT DCA SCHOOL 


Your response to the following items will help modify 

the IDCTT program to more specifically address your Command’s 

training goals. This survey is designed to assess how instructors rate 
different aspects of the IDCTT Trainer effectiveness. There are also 
short answer questions where you can express your opinion on the system. 
Please answer all of the questions completely, explaining your answers 
thoroughly. Use the back of the questionnaire if additional answer space 
1s necessary. 


1. Approximately how many hours did you spend assisting students with the 
rerr Trainer? 
Hours 


2. From an instructor’s perspective, rate how easily the IDCTT Trainer 
allowed you to instruct students in damage control simulator training. 


Very Very 
Easy Neutral Difficult 
a 2 3 4 5 6 7 8 9 10 dbl 


3. Rate how realistic the IDCTT Trainer depicts damage control training 
compared to actual shipboard damage control. 


Very Very 
Realistic Neutral Artificial 
1 2 3 4 5 6 a 8 9 10 dal 


4. Rate the extent to which you would like to see the IDCTT Trainer used 
as a permanent aid for simulation training at the DC School. 


Very Very 
Much Neutral Little 
uf 2 3 4 5 6 a 8 9 LG 11 


5S. Rate the students reaction (positive or negative) to the IDCTT Trainer 
meomoamernstructional aid. 


Very Very 
Positive Neutral Negative 
il 2 3 4 5 6 i 8 9 10 aT 


6. Rate how beneficial the IDCTT Trainer would be as an installed 
ehijpreoard training tool. 


Very Very 
Beneficial Neutral Detrimental 
al 2 3 4 5 6 ay 8 9 10 dal 
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7. What aspects did you like about the IDCTT for teaching damage control 
problems? 


8. What problems did you encounter while using the IDCTT as an 
instructionalwald- 


9. What aspects of the IDCTT would you like to see changed? 


10. What benefits do you envision from the use of IDCTT at the Damage 
Control! Schoo, 
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STUDENT GRADE SHEET FOR IDCTT AND DC CENTRAL 


TRAINER SCENARIOS 


STUDENT’S NAME (DCA) DATE 


GROUP MEMBER NAMES 


INSTRUCTOR’S NAME 


METHOD (CIRCLE ONE) : Loe rs DC CENTRAL TRAINER 


The following is a grade sheet for IDCTT and DC Central Trainers. The 
results will be used for research purposes only! Grades assigned using 
this evaluation form will not effect student grades in any capacity. This 
performance evaluation will be used for the sole purpose of comparing the 
IDCTT and DC Central Trainer methods. 

Maximum points allowed for each topic are indicated in parentheses 
next to each topics. Partial credit may be awarded when actions are taken 
for a topic area but not completely or correctly executed. For example, if 
fire boundaries were successfully set on 3 of 5S spaces requiring this 
action, the student would receive 6 out of a possible 10 points. The 
instructor will determine what percentage of the topic area was 
accomplished and assign points accordingly. 


1. DCA ensured zebra was set on firemain (5 poincs) 

2. Investigators sent out to investigate (5 points) 
ruptured piping when pressure indications 
suggested a rupture. 

3. Maintained firemain pressure through (> Po1rncs) 

lsolating damaged spots and firepump 

management. 

4. Investigators sent out to investigate (Sapoints) 
all fire and flooding alarms. 

5. Fire boundaries set around compartments (10 points) 
on fire. : 

6. Smoke boundaries set around compartments (S>poincs) 
en Lire. 

7. Flooding boundaries set around flooding/ (10 points) 
flooded compartments. 

8. Priority focused on fires/floods near (10 poines) 


key spaces (1.e., magazines, radar equipment, CIC). 
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oe 


He. 


AAs 


eee 


Loe 


14. 


Absre 


Efforts made to fight fires with assets 
available. 


Efforts made to isolate ruptures 
(non firemain 1.67, e@nil) water, sue 
lube oil). 


Attempts made to repair damage when 
identified (1.e., Ge@ptured chill water, 
ruptured firemain). 


When prompted for action did the DCA 
respond in an effective and timely 


manner (from repair lockers, bridge, CIC). 


Respond to prompts from CSMC on loss of 


chill water or overheating arrays/radars. 


Maintain the big picture of the 
damage and circumstances that he was 
combating. 


Keep the Commanding Officer/bridge 


informed on the status of damage control 
effortce 


TOTAL POINTS SCORED 


Comments: 
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(5 posits} 


(5 pointes) 


(5 pointes) 


(5 points) 


(10 points) 


(lO”"poimes) 


(5 pointes 


(100 points) 


STUDENT IDCTT VS DC CENTRAL TRAINER COMPARISON 
SURVEY 


NAME DATE 


RANK 


Your answers to the following questions will help improve the training 


you receive in Damage Control Central simulators. This survey will 
provide information for comparing the effectiveness of the IDCTT Trainer 
and DC Central Trainer as a training aids. This survey is designed to 


determine your opinion on the ability of the IDCTT and DC Central Trainers 
to provide quality training. If you have any comments that the numerical 
scale do not address, please write your comments on the back of the 
survey. Upon completion, please return this form to your instructor. 


1. Rate which method provides a more realistic model of an actual 
shipboard damage control environment. 


pe TT Neutral Dcc Trainer 
al Zz. 3 4 5 6 7 8 9 sme) at 


2. Rate which method provides the ability for the instructor to provide 
the most complete post scenario debriefs. 


PoCTT Neutral Dec Trainer 
d Z 3 4 5 6 7 8 9 10 11 


3. Rate which method produced the greatest level of stress while 
performing the damage control scenario. 


LoeTT Neutral Dec Trainer 
a eo, = 4 5 6 q 8 9 10 ale 
4. Rate which system enabled the instructors to monitor student 


performance more closely. 


rmoeTT Neutral Dpcc Trainer 
uh Z 3 4 5 6 7 8 s) 10 11 


5. Rate which method will better prepare you for the actual casualties 
that might encounter while onboard a ship. 


rei T Neutral Dcc Trainer 
1 2 3 4 5 6 -! 8 9 i @) ah 


6. Rate which method provides an easier means to take action on scenario 
problems (i.e., sound powered phones, DC plate plotting and hardwired 
alarm panels vise touchscreen inputs and computer monitor alarm panels) 


Pocrr Neutrai DccC Trainer 
al Z 3 4 5 6 7 8 9 10 il 
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7. Rate which method provides scenario information in a manner most 
closely resembling shipboard emergency situations. 


IDCTT Neutral DCC Trainer 
1 2 3 4 Ss 6 a 8 9 10 dio 


8. Rate which method is more effective in teaching damage control skills 
necessary to combat damage control problems 


TPDECTT Neutral Dcc Trainer 
al g, 3 4 5 6 7 8 9 LO Ale 


9. Rate which system promoted greater learning in the amount of time 
allocated. 


TDCTT Neutral Dcc Trainer 
il 2 3 4 S 6 a 8 9 10 lel 


10. If you had access to one method of instruction rate which system you 
would prefer. 


LDETT Neutral Dcc Trainer 
ib 2 3 4 5 6 7 8 9 10 Aye 
11. Rate which method inspires you to:perform to the best of your ability. 


IDCT? Neutral Dcc Trainer 
i Z 3 4 5 6 fi 8 9 10 lal 
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SCENARIO TOPICS RANKING 


NAME DATE 
fois BANKING IS °FOR (CIRCLE ONE): IDCTT SCENARIO 
DC CENTRAL TRAINER SCENARIO 


The following thirteen topics summarize the principal actions 
necessary to successfully complete a damage control problem. The 
instruction method to which you were just exposed emphasized each of the 
following topics to varying degrees. In the blank space provided to the 
left of each topic, rank each topic from 1 to 7 based on how you felt 
the method (IDCTT or DC Central Trainer) emphasized the importance of each 
topic to complete the DC problem. Use the following criteria to express 
Wowk Opilnilon: 


Not At All Extremely 
Important Neutral Important 
1 Z 32 4 5 6 a 


NOTE: Base your rankings on what the scenario emphasized and not on what 
you think should be emphasized. 

Maintain effective communications 

Keep the chain of command informed 

Ensure Zebra is set 

Restore vital systems (1.e., mechanical, chill water, ventilation) 
Manage Damage Control Central 

Isolate damage (smoke, fire and flood boundaries) 

Confirm proper manned and ready reports 

Restore Firemain 

Manage personnel casualties and evacuations 

Locate damage 

Prioritize casualties 

Coordinate firefighting and repair teams 


Isolate explosive hazards (1.e., magazines, fuel tanks) 
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APPENDIX B STUDENT AND INSTRUCTOR ESSAY RESPONSES 


IDCTT Trainer Survey Question 5 


Briefly explain why any of the below listed IDCTT Trainer features caused 
you difficulty while operating the system. 


A. 


Inputing information with the touchscreen monitor 


Trying to input information rapidly with large fingers on a relatively 
small screen. 


More distance needed between touchscreen buttons to avoid acei1Ga-nee 
TMpuUts - 


Touchscreen needs to be more sensitive. 

Touchscreen did not always take information first time it was entered. 
Too slow when ordering actions. 

Touchscreen responded too slowly. 

Touchscreen looked like it activated only to find out that le .amemweee 
Difficult to press correcce bucecen-- 

Missed information while typing in orders on the touchscreen. 

Screen did not respond promptly to inputs. 

Had difficultly getting to appropriate menu. 

Did not know if orders were accepted. 

Difficult to input daw 

DiEfFICUlE anputing aGantormaeren: 

Did not take information inputs. 

Hard to get used to touchscreen. 


Due to the speed of the scenario it was difficult to input necessary 
information in time given. 


= Ol 


Unfamiliarity with the inputing system and the ship involved made it 
Cer ricuilt toe enter iomnf)SOrmation. 


Touchscreen was not very sensitive, I usually had to hit the screen 2 
times to activate the option. 


Sometimes you had to touch the screen two or three times to input 
information. 


Events were overwhelming at times due to the slow data input rate. 
It is easy to get overwhelmed by incoming information because you can 
not give immediate responses to the reports, so incoming reports mount 


up before the DCA can tell the computer what to do. 


Entering data was difficult due to unfamiliarity with the sequencing 
logic and symbology. It got easier as the program progressed. 


Some options were difficult to discern or are not activated. 


Understanding audio reports 


Background noise sounds like a report because it keeps saying, 
Er fo. . ” 


Background noise contained clearly erroneous information. 


Could not ask for repeat of volce messages. 


minding DC plate information 


Never had seen DDG-51 chill water or compartment DC Plates, which led to 
problems isolating systems. 


Not used to DDG-51 Class. 
Memelates difficult to understand. 
@eulla not find the valve numbers I wanted to close. 


Could not locate the information needed. 


Speed or volume of information presented 


Too fast for a beginner, need to be able to adjust speed with which 
information is given. 


Speed of information presented overloads the DCA at first. 
Too many printed messages. 
Lost track of audio reports and tried to key on important issues. 


Macsive volume of data was difficult to track and remember but this 
seems to make the simulation more realistic. 


oA 


E. Damage control alarm panel display 


No responses. 


F. Firemain panel and firemain valve and pump operations 
The firemain panel is busy and it goes by fast. 
Firemain screen is too small, can’t read print. 


Firemain panel, especially, fire pumps were difficult to read and 
understand. 


G. Cther 


The initial five to ten minutes of the first run was just system 
familiarization. 


Not able to find out the status of the chill water system. 

Big trouble learning what actions the repair lockers, etc, were doing on 
their own and what I needed to direct. Also, what I had to tell lockers 
to get things done. 

Lack of experience (my problem). 

Tried to order a COV to Repair V when it was in the Repair III area. 
Computer just said, "review your last order" and did not say that the 
valve was not in their area. Locker should say, "That is in Repair III 


areavsorr ae 


The program will not accept spaces to investigate unless damage is 
reported in the space. 


Unable to give space name to locker and let the locker look up the fire 
and flooding boundaries. 


IDCTT Trainer Survey Question 8 


How can the touchscreen control panel be improved? 
T €hink 1 iS sfane- 
The screen, at times, does not respond well. 
Larger number pad on touchscreen, use keyboard instead. 
Put more space in between buttons. 


Increase touchscreen sensitivity. 


a 


Better touchscreen sensitivity. 

Practice. 

Make touchscreen more sensitive to the touch and quicker to respond. 
Auto advance through “setting boundaries" after enter is pushed. 
Teach the users the proper method of touching the screen. 


Program a five to seven second pause into the system to input 
information. Voice activated commands would be best. 


More sensitive with quicker response time. 

Faster button response to the touch. 

Touchscreen does not always “take" information when you push the area, 
other times you have to keep your finger on it for a long time before it 
ececepes YOur input. 

Difficult to enter information on screen, improve screen sensitivity. 
Add an audio bell to indicate that the inputed information was accepted. 
Use mouse control to increase speed. 

Bapuc information through the keyboard only. 

Switch to a mouse. 

Switch to voice activation. 

Make status and repair keys give better information. 

Use keyboard vice screen keypad. 

Not all actions are acknowledged the same way causing multiple inputs. 
Improve system so screen does not freeze up. 

Improve sensitivity. 


Wewee activation of DCA orders. 


IDCTT Trainer Survey Question 10 


What problems did you encounter while using the IDCTT Trainer? 
The Command Console number pad is slow in responding to inputs. 
Unfamiliarity with DDG-51 systems. 


No method for confirming actions ordered were taken. While inputing 
information into the system, I accidentally cancelled the input. 


Inputing information through the touchscreen and background noise. 


= Ae) 


Dirty screen causing the computer to act funny. We cleaned the screen 
and everything was fine. 


Learning what functions each button option would do. 


Losing information inputed through the screen because I did not press 
tT] OK tu : 


Initial system familiarization. 
System froze to reload printer paper and some kind of memory error. 


Coordinating all reports and correlating them to the various plots and 
displays. 


No way of asking the status of boundaries being set. 
Focusing energy on inputing information rather then the battle problem. 
Unfamiliarity with the system, need more time to practice. 


Lost control of the situation, typed in flooding boundaries when I 
wanted to set fire boundaries. 


Lack of knowledge of chill water system killed me. 

Inexperienced in making reports to the commanding officer. 
InpuULINGgmInteorumaclton Imeo the Loucmscreen. 

Printer stalled during the battle problem. 

System would lock-up for a couple of seconds for unknown reason. 


Message blanks are good but too much information is given in the amount 
of time allotted to read them. 


No clear means of reporting information to the Captain/Bridge. 
Confusion caused by background noise that gave incorrect reports. 
Lack of experience with equipment. 

Finding the correct keys co seucn. 

Knowing what was done automatically. For example during beginning 
sequence, zebra on Firemain 1s reported set, but I had to order the 


valve closed. 


Learning what to press to make reports to the Commanding Officer was 
di tei ren ie. 


Lack of knowledge of how to use it. 
Touchscreen sensitivity and becoming familiar with the computer. 


Knowledge of the repair locker capabilities and what I had to tell them 
to carry out what I wanted, ex: fight for fighting fires. 


Sometimes correct inputs were not understood by the computer (lie, fight 
fires O-1 level was accepted but actual compartment number was not). 
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The magazine blew up even though I lit off the deluge system. 
Lack of familiarity with the system. 
Dirty screen caused faulty input responses. Unfamiliarity with command 


and control functions caused some confusion early on but became less of 
a problem as experience increased. 


IDCTT Trainer Question 11 


What aspects of the IDCTT Trainer did you like the most? 
Trainer seems realistic. 
Interactive aspects. 
Clarity of information received. 


Realistic flow of events and speed of damage. DCA gets an actual test 
rather than Knowing what will happen. 


Amount of stress induced by the system. 

Good system for training, don’t have to go to GQ to operate. 
Audio was excellent and message blanks provided added realism. 
Poglaty for one person to train without the ship going to GQ. 
Being able to repeat a scenario until you get it right. 

Audio and video messages. 

Visually realistic representation of DC member images. 

Pace and presentation of information. 

Wery realistic. 

Clear information through printer messages. 

Printed DC messages. 

Video images of people telling you information. 

Fast and furious flow of information seems fairly realistic. 
Problems cascade if appropriate actions are not taken. 

Fast paced audiovisual displays make you think quickly. 


Repetition of the same scenario allows you to learn the proper sequence 
of actions to combat the damage. 


Good training aid that focuses on the DCAs skill. 


Visual display of actions taking place. 


ee) 


Real life feeling. 
Speed and reality of scenario. 


Realistic, fast paced, you as a student became very involved. I was 
impressed. 


Stresses the DCA and starts preparing him for how he needs to think in 
DE-“Central 


Stress technical aspects of DC, such as boundaries and reenforces them 
much better then the DC Central Trainer. 


Video display makes it look very realistic. 

Sound effects and voice reports. 

Video and audio effects. 

One man trainer does not require any ship’s assets to initiate. 
High intensity, realistic locker actions, no stupid locker leaders. 


It did not take a whole watch team to get training. Individual training 
is a lot easler to schedule in real life. 


The fast paced multiple casualties. 


Lets you see the consequences of your actions, le. securing chill water 
makes you lose you Aegis system. 


Very realistic and stressful. Seems to fairly represent an actual 
damage scenario. It forces you to prioritize casualties an allot 
resources accordingly. 


The background noise, the pictures of personnel talking to you, the many 
options, this all made it very realistic. 


IDCTT Trainer Survey Question 12 


What aspects of the IDCTT Trainer did you like the least? 
Waste of paper in repeatedly printing out DC chits. 
Only one scenario. 

Pace of scenario. 

Difficulty inpwieing amteunmacien: 
Diff£1eulty while anput ing sin ternat ier. 
Firemain screen 1S too small. 


Comments from post scenario reasons for kill point activation are too 
general. 
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Trying to use the DC Plates while operating the computer was difficult. 


Potential for other damage control team members, who do not have this 
system, to go untrained. 


Not being able to tell someone verbally what to do. 

Not all options are clearly expressed in the Command Console’s menu. 
Touchscreen. 

Not able to order three things at once. 

Firemain control panel. 

Chill water seems to be a big emphasis, and it should be, but some 
amount of ship specific knowledge is required to do well on the 
Simulator, so this facet is pretty pointless unless you are going toa 
mpG-51. 

Some options on the menu could not be initiated. 

Need more drill scenarios. 

Need more ship classes. 

Did not drill the entire Damage Control Central watch team. 

Limited options available. 

Need different scenarios. 

Does not train how to handle DC Central watch team. 

The DCAs load is more because he has to type information into the 
computer while in the DC Central Trainer he is less loaded because he 


talks and directs. 


Inaccurate information in background noise is distracting and 
misleading. 


Method of issuing orders. 

Definitely a game. 

DCA does not sit at a computer, he stares at charts. If the DCA could 
stand and plot and order actions it would be better. This could be 
solved by having the DCA stand and have a computer operator input his 
orders. 


Was not able to jumper systems. 


Data input slow. Touchscreen keyboard locked up a few times. Sometimes 
hard to get computer to do what you want it to do. 


It takes a relatively long time to input data. Even if one is familiar 
with keyboards, data input is slow compared to voice commands. 


Could respond more quickly ina reai locker. Did not like having to 
input every small detail, but it did make you think things out. 


oF 


Only one scenario 
IDCTT Instructor Survey Question 7 


What aspects did you like about the IDCTT Trainer for teaching damage 
control problems? 


Emphasized the basic sequence of damage control problems evolution 
(l.e., set boundaries, mechanically and electrically isolate, fight 
fires w Cec.) 


It objectively allows for multiple paths to be initiated, which is 
difficult to replicate inthe sPesCentra be biaaage 


Ability to present identical scenarios to each DCA student. 


The ability to objectively critique the DCA’s actions (1.e., pee veme. 
the "No, I didn’ "22323 Yes, seu didsaiceqnaric ~ 


Printed message blanks. 


Realism of noise and confusion that would occur during a damage control 
problem. 


The realism, in the fleet it is very hard to reproduce the realistic 
pressure involved in DC scenarios. 


The stress level that it creates. 
Imposes very realistic level of stress on student, particularly through 
the amount of informant given, the manner in which the information is 


given, and in the time in which it is given. 


Reinforces basic and advanced damage control concepts (some with 
immediate feedback, some with delayed feedback, li.e., ending program). 


IDCTT Instructor Survey Question 8 
What problems did you encounter while using the IDCTT as an 
instructional aid? 


The program did not run in real time. It moved more like a video game 
than an actual scenario. 


Student familiarity. 

Certain situations could only be corrected by one, and only onewacenoe 
Real time versus simulator time. 

Speed of input, the touchscreen frustrated students. 

Touchscreen slowed DCA's responses. 


Students unfamiliar with the DDG-51 Class platform. 
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Fairly extensive pre-brief in classroom required to fully prepare 
Students ror I[DeCTT. 


IDCTT interface still a bit limiting (time wise). 
IDCTT locks up occasionally (very seldom now). 


Problems with inactive buttons on touchscreen monitor (having to explain 
Emey WO noe function) : 


IDCTT Instructor Survey Question 9 


What aspects of the IDCTT Trainer would you like to see changed? 


CSMC should have its own repair team that isolates and corrects chill 
water problems. The ship should not rely heavily on the DCA for Combat 
System casualties. 


The DCA has to prompt too many actions. Fire and flooding boundaries, 
and isolation would be reported to the DCA rather than ordered by the 
DEA. 


Increase the avenues in which the DCA may approach solutions to the 
scenario. 


Provide feedback from the lockers in response to incorrect or improper 
orders. 


Change touchscreen to voice recognition. 


Obviously, in the future, IDCTT will ideally have multiple scenarios and 
Multiple platforms to choose from. 


More versatility. The IDCTT Trainer needs to have more scenarios and if 
the DCA could program IDCTT himself, it would add more. 


Scenario made a little more robust by adding more options for the 
students to initiate various orders. 


More scenarios. 


IDCTT Instructor Survey Question 10 


What benefits do you envision from the use of the IDCTT Trainer at the 
Damage Control School? 


Pie iDeCT)T can be “connected” to an EOOW, OOD and CICWO Trainer to do an 
integrated ship training evolution, this will be a vehicle for us. 


May be used as a final simulation to objectively determine a students’ 
ability to operate under stressful conditions. 


oo 


The ability to better prepare a student for the 
associated with live DC scenarios in the fleet. 


It better prepares DCAs to face the stresses of 
disasters. 


Better preparing the DCA for the real situation. 
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pressures and problems 


real shipboard 


APPENDIX C WILCOXON SIGNED~RANK TEST CALCULATIONS 


Null Hypothesis: H,: The population distribution for the 
IDCTT Trainer and DC Central Trainer performance scores are 
identical. 


Alternative Hypothesis: H,: The population relative frequency 
distribution of the IDCTT Trainer performance scores is 
shifted to the right of the DC Central performance scores. 


Data: 
ig = Rank sum of the positive differences 
= 350 
Ay = Rank sum of the negative differences 
= i035 
n — el 
Note: Table contains the difference and rank scores for 32 
student participants 
Test Statistic: z = 1.99 (calculated from below equation) 


P= T*-[n(nt1) /4) 
Ynt(n+l) (2n+1) 724 


Rejection Region: Reject H, if z 2 Z, 
Z = 1.99 
Ze = 1.96 at the .025 significance level 


Conclusion: Reject the null hypothesis at the .025 significance 
level. The distribution of performance scores for the IDCTT 
Trainer is higher then the distribution of performance scores 
for the DC Central Trainer. 
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TABLE 17: STUDENT TRAINER SCORES 


| Student DC Student LDC DC 
Central Central 
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TABLE 18: IDCTT AND DC CENTRAL TRAINER WILCOXON RANKINGS 


— 


Student Difference 
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